allow disabling Ruleset

README.md

@@ -10,8 +10,6 @@
 
 Freedom of information is an essential pillar of democracy and informed decision-making. While media organizations have legitimate financial interests, it is crucial to strike a balance between profitability and the public's right to access information. The proliferation of paywalls raises concerns about the erosion of this fundamental freedom, and it is imperative for society to find innovative ways to preserve access to vital information without compromising the sustainability of journalism. In a world where knowledge should be shared and not commodified, paywalls should be critically examined to ensure that they do not undermine the principles of an open and informed society.
 
-Certain sites may display missing images or encounter formatting issues. This can be attributed to the site's reliance on JavaScript or CSS for image and resource loading, which presents a limitation when accessed through this proxy. If you prefer a full experience, please concider buying a subscription for the site.
-
 > **Disclaimer:** This project is intended for educational purposes only. The author does not endorse or encourage any unethical or illegal activity. Use this tool at your own risk.
 
 ### Features
@@ -23,14 +21,24 @@ Certain sites may display missing images or encounter formatting issues. This ca
 - [x] Fetch RAW HTML
 - [x] Custom User Agent
 - [x] Custom X-Forwarded-For IP
-- [x] [Docker container](https://github.com/kubero-dev/ladder/pkgs/container/ladder)
+- [x] [Docker container](https://github.com/kubero-dev/ladder/pkgs/container/ladder) (amd64, arm64)
 - [x] Linux binary
 - [x] Mac OS binary
 - [x] Windows binary (untested)
-- [x] Removes most of the ads (unexpected side effect)
+- [x] Removes most of the ads (unexpected side effect ¯\_(ツ)_/¯ )
 - [x] Basic Auth
 - [x] Disable logs
 - [x] No Tracking
+- [x] Limit the proxy to a list of domains
+- [x] Expose Ruleset to other ladders
+- [ ] Optional TOR proxy
+- [ ] A key to share only one URL
+- [ ] Fetch from Google Cache if not available
+
+### Limitations
+Certain sites may display missing images or encounter formatting issues. This can be attributed to the site's reliance on JavaScript or CSS for image and resource loading, which presents a limitation when accessed through this proxy. If you prefer a full experience, please concider buying a subscription for the site.
+
+Some sites do not expose their content to search engines, which means that the proxy cannot access the content. A future version will try to fetch the content from Google Cache.
 
 ## Installation
 
@@ -70,11 +78,15 @@ curl -X GET "http://localhost:8080/api/https://www.example.com"
 ### RAW
 http://localhost:8080/raw/https://www.example.com
 
+
+### Running Ruleset
+http://localhost:8080/ruleset
+
 ## Configuration
 
 ### Environment Variables
 
-| Variable | Description | Default |
+| Variable | Description | Value |
 | --- | --- | --- |
 | `PORT` | Port to listen on | `8080` |
 | `PREFORK` | Spawn multiple server instances | `false` |
@@ -83,10 +95,45 @@ http://localhost:8080/raw/https://www.example.com
 | `USERPASS` | Enables Basic Auth, format `admin:123456` | `` |
 | `LOG_URLS` | Log fetched URL's | `true` |
 | `DISABLE_FORM` | Disables URL Form Frontpage | `false` |
-| `RULES_URL` | URL to a ruleset file | `https://raw.githubusercontent.com/kubero-dev/ladder/main/ruleset.yaml` |
+| `FORM_PATH` | Path to custom Form HTML | `` |
+| `RULESET` | URL to a ruleset file | `https://raw.githubusercontent.com/kubero-dev/ladder/main/ruleset.yaml` or `/path/to/my/rules.yaml` |
+| `EXPOSE_RULESET` | Make your Ruleset available to other ladders | `true` |
+| `ALLOWED_DOMAINS` | Comma separated list of allowed domains. Empty = no limitations | `` |
+| `ALLOWED_DOMAINS_RULESET` | Allow Domains from Ruleset. false = no limitations | `false` |
+
+`ALLOWED_DOMAINS` and `ALLOWED_DOMAINS_RULESET` are joined together. If both are empty, no limitations are applied.
 
 ### Ruleset
 
 It is possible to apply custom rules to modify the response. This can be used to remove unwanted or modify elements from the page. The ruleset is a YAML file that contains a list of rules for each domain and is loaded on startup
 
-See in [ruleset.yaml](ruleset.yaml) for an example.
+See in [ruleset.yaml](ruleset.yaml) for an example.
+
+```yaml
+- domain: www.example.com
+  regexRules:
+    - match: <script\s+([^>]*\s+)?src="(/)([^"]*)"
+      replace: <script $1 script="/https://www.example.com/$3"
+  injections:
+    - position: head # Position where to inject the code
+      append: | 
+        <script>
+          window.localStorage.clear();
+          console.log("test");
+          alert("Hello!");
+        </script>
+- domain: www.anotherdomain.com # Domain where the rule applies
+  paths:                        # Paths where the rule applies
+    - /article
+  googleCache: false            # Search also in Google Cache
+  regexRules:                   # Regex rules to apply
+    - match: <script\s+([^>]*\s+)?src="(/)([^"]*)"
+      replace: <script $1 script="/https://www.example.com/$3"
+  injections:
+    - position: .left-content article .post-title # Position where to inject the code into DOM
+      replace: | 
+        <h1>My Custom Title</h1>
+    - position: .left-content article # Position where to inject the code into DOM
+      prepend: | 
+        <h2>Suptitle</h2>
+```

cmd/main.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	_ "embed"
+	"fmt"
 
 	"ladder/handlers"
 	"log"
@@ -9,6 +10,7 @@ import (
 	"strconv"
 	"strings"
 
+	"github.com/akamensky/argparse"
 	"github.com/gofiber/fiber/v2"
 	"github.com/gofiber/fiber/v2/middleware/basicauth"
 	"github.com/gofiber/fiber/v2/middleware/favicon"
@@ -19,10 +21,32 @@ var faviconData string
 
 func main() {
 
-	prefork, _ := strconv.ParseBool(os.Getenv("PREFORK"))
+	parser := argparse.NewParser("ladder", "Every Wall needs a Ladder")
+
+	p := os.Getenv("PORT")
+	if os.Getenv("PORT") == "" {
+		p = "8080"
+	}
+	port := parser.String("p", "port", &argparse.Options{
+		Required: false,
+		Default:  p,
+		Help:     "Port the webserver will listen on"})
+
+	pf, _ := strconv.ParseBool(os.Getenv("PREFORK"))
+
+	prefork := parser.Flag("P", "prefork", &argparse.Options{
+		Required: false,
+		Default:  pf,
+		Help:     "This will spawn multiple processes listening"})
+
+	err := parser.Parse(os.Args)
+	if err != nil {
+		fmt.Print(parser.Usage(err))
+	}
+
 	app := fiber.New(
 		fiber.Config{
-			Prefork: prefork,
+			Prefork: *prefork,
 		},
 	)
 
@@ -49,15 +73,13 @@ func main() {
 	}
 
 	app.Get("/", handlers.Form)
+	app.Get("ruleset", handlers.Ruleset)
 
 	app.Get("raw/*", handlers.Raw)
 	app.Get("api/*", handlers.Api)
+	app.Get("ruleset", handlers.Raw)
 	app.Get("/*", handlers.ProxySite)
 
-	port := os.Getenv("PORT")
+	log.Fatal(app.Listen(":" + *port))
-	if os.Getenv("PORT") == "" {
-		port = "8080"
-	}
-	log.Fatal(app.Listen(":" + port))
 
 }

go.mod

@@ -11,6 +11,7 @@ require (
 
 require (
 	github.com/PuerkitoBio/goquery v1.8.1
+	github.com/akamensky/argparse v1.4.0
 	github.com/andybalholm/brotli v1.0.6 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/google/uuid v1.4.0 // indirect

go.sum

@@ -1,5 +1,7 @@
 github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
 github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
+github.com/akamensky/argparse v1.4.0 h1:YGzvsTqCvbEZhL8zZu2AiA5nq805NZh75JNj4ajn1xc=
+github.com/akamensky/argparse v1.4.0/go.mod h1:S5kwC7IuDcEr5VeXtGPRVZ5o/FdhcMlQz4IZQuw64xA=
 github.com/andybalholm/brotli v1.0.6 h1:Yf9fFpf49Zrxb9NlQaluyE92/+X7UVHlhMNJN2sxfOI=
 github.com/andybalholm/brotli v1.0.6/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
 github.com/andybalholm/cascadia v1.3.1 h1:nhxRkql1kdYCc8Snf7D5/D3spOX+dBgjA6u8x004T2c=

handlers/form.go

@@ -1,192 +1,31 @@
 package handlers
 
 import (
+	_ "embed"
+	"log"
 	"os"
 
 	"github.com/gofiber/fiber/v2"
 )
 
+//go:embed form.html
+var formHtml string
+
 func Form(c *fiber.Ctx) error {
 	if os.Getenv("DISABLE_FORM") == "true" {
 		c.Set("Content-Type", "text/html")
 		c.SendStatus(fiber.StatusNotFound)
 		return c.SendString("Form Disabled")
 	} else {
+		if os.Getenv("FORM_PATH") != "" {
+			dat, err := os.ReadFile(os.Getenv("FORM_PATH"))
+			if err != nil {
+				log.Println("ERROR: unable to load custom form", err)
+			} else {
+				formHtml = string(dat)
+			}
+		}
 		c.Set("Content-Type", "text/html")
-		return c.SendString(html)
+		return c.SendString(formHtml)
 	}
 }
-
-const html = `
-<!DOCTYPE html>
-<html>
-<head>
-    <meta charset="UTF-8">
-    <title>Ladder</title>
-    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css">
-</head>
-    <style>
-        body {
-            background-color: #ffffff;
-        }
-
-        header h1 {
-            text-transform: uppercase;
-            font-size: 70px;
-            font-weight: 600;
-            color: #fdfdfe;
-            text-shadow: 0px 0px 5px #7AA7D1, 0px 0px 10px #7AA7D1, 0px 0px 10px #7AA7D1,
-                0px 0px 20px #7AA7D1;
-        }
-        .logo-title {
-            font-family: 'Arial', sans-serif;
-            font-size: 2rem;
-            color: #fff;
-            margin-bottom: 20px;
-        }
-        .logo {
-            text-align: center;
-        }
-        .github-corner {
-            animation: octocat-wave 560ms ease-in-out;
-            position:absolute;
-            top:0;
-            right:0;
-        }
-    </style>
-</head>
-<body>
-    <a href="https://github.com/kubero-dev/ladder">
-        <div class="github-corner" aria-label="View source on GitHub">
-            <svg
-                xmlns:svg="http://www.w3.org/2000/svg"
-                xmlns="http://www.w3.org/2000/svg"
-                xmlns:xlink="http://www.w3.org/1999/xlink"
-                version="1.1"
-                width="146"
-                height="146"
-                id="svg2">
-                <defs
-                    id="defs8">
-                    <filter
-                    height="1.096"
-                    y="-0.048"
-                    width="1.096"
-                    x="-0.048"
-                    style="color-interpolation-filters:sRGB"
-                    id="filter6">
-                    <feGaussianBlur
-                        stdDeviation="3"
-                        id="feGaussianBlur4" />
-                    </filter>
-                </defs>
-                <path
-                    d="M 152,140 6,-6 H 48 L 152,98 Z"
-                    style="opacity:0.8;filter:url(#filter6)"
-                    id="path10" />
-                <path
-                    d="M 146,134 12,0 h 42 l 92,92 z"
-                    style="fill:#007200"
-                    id="path12" />
-                <g
-                    aria-label="Fork me on GitHub"
-                    transform="rotate(45)"
-                    style="font-family:Collegiate;fill:#ffffff"
-                    id="g42">
-                    <path
-                    d="m 53.643,-19.486 c 0,0.688 -0.016,1.2 -0.064,1.504 h 2.08 c -0.048,-0.32 -0.064,-0.8 -0.064,-1.424 v -3.344 h 1.76 c 0.416,0 0.736,0.016 0.944,0.048 v -1.76 c -0.24,0.032 -0.592,0.048 -1.088,0.048 h -1.616 v -2.496 h 1.936 c 0.56,0 0.944,0.016 1.184,0.048 v -1.792 h -5.136 c 0.048,0.272 0.064,0.784 0.064,1.504 z"
-                    id="path14" />
-                    <path
-                    d="m 62.424,-17.87 c 1.008,0 1.776,-0.368 2.272,-1.088 0.432,-0.624 0.656,-1.472 0.656,-2.544 0,-2.416 -0.976,-3.616 -2.928,-3.616 -1.968,0 -2.96,1.2 -2.96,3.616 0,1.072 0.224,1.936 0.656,2.56 0.512,0.72 1.28,1.072 2.304,1.072 z m -0.016,-5.68 c 0.496,0 0.816,0.24 0.976,0.704 0.096,0.272 0.144,0.72 0.144,1.344 0,0.64 -0.048,1.088 -0.144,1.36 -0.16,0.464 -0.48,0.688 -0.976,0.688 -0.496,0 -0.816,-0.24 -0.976,-0.704 -0.096,-0.272 -0.144,-0.72 -0.144,-1.344 0,-0.624 0.048,-1.072 0.144,-1.344 0.16,-0.464 0.48,-0.704 0.976,-0.704 z"
-                    id="path16" />
-                    <path
-                    d="m 68.293,-17.982 c -0.032,-0.24 -0.048,-0.64 -0.048,-1.184 v -3.888 c 0.352,-0.304 0.672,-0.464 0.976,-0.464 0.224,0 0.48,0.096 0.752,0.288 v -1.808 c -0.224,-0.08 -0.432,-0.128 -0.624,-0.128 -0.448,0 -0.832,0.192 -1.152,0.56 v -0.48 h -1.744 c 0.032,0.192 0.048,0.544 0.048,1.04 v 4.976 c 0,0.512 -0.016,0.88 -0.048,1.088 z"
-                    id="path18" />
-                    <path
-                    d="m 72.857,-17.982 c -0.032,-0.24 -0.048,-0.64 -0.048,-1.184 v -2.448 l 1.472,2.816 c 0.208,0.384 0.32,0.656 0.368,0.816 h 1.872 l -2.352,-4.416 2.144,-2.752 h -2.064 c -0.08,0.176 -0.192,0.352 -0.352,0.56 l -1.088,1.44 v -4.496 c 0,-0.464 0.016,-0.8 0.048,-1.008 h -1.824 c 0.032,0.192 0.048,0.544 0.048,1.04 v 8.544 c 0,0.512 -0.016,0.88 -0.048,1.088 z"
-                    id="path20" />
-                    <path
-                    d="m 85.08,-24.478 c -0.384,-0.432 -0.896,-0.656 -1.52,-0.656 -0.416,0 -0.864,0.192 -1.328,0.56 v -0.512 l -1.76,-0.016 c 0.032,0.176 0.048,0.544 0.048,1.12 v 4.992 c 0,0.496 -0.016,0.832 -0.048,1.008 h 1.856 c 0,-0.064 -0.048,-0.64 -0.048,-1.008 v -3.984 c 0.304,-0.288 0.608,-0.432 0.928,-0.432 0.656,0 0.864,0.416 0.864,1.76 l -0.016,2.16 c 0,0.656 -0.032,1.168 -0.08,1.504 h 1.92 c -0.048,-0.256 -0.064,-0.752 -0.064,-1.472 v -2.192 c 0,-0.56 -0.048,-1.056 -0.144,-1.504 0.208,-0.176 0.544,-0.256 0.976,-0.256 0.64,0 0.96,0.592 0.96,1.76 v 2.16 c 0,0.656 -0.032,1.168 -0.08,1.504 h 1.904 c -0.048,-0.256 -0.064,-0.752 -0.064,-1.472 v -2.192 c 0,-0.96 -0.176,-1.744 -0.512,-2.368 -0.432,-0.752 -1.056,-1.12 -1.888,-1.12 -0.736,0 -1.376,0.224 -1.904,0.656 z"
-                    id="path22" />
-                    <path
-                    d="m 95.905,-20.99 c 0.032,-0.304 0.048,-0.624 0.048,-0.992 0,-0.944 -0.224,-1.696 -0.656,-2.256 -0.464,-0.592 -1.136,-0.896 -2.016,-0.896 -0.896,0 -1.6,0.368 -2.112,1.088 -0.464,0.656 -0.688,1.456 -0.688,2.432 0,1.136 0.272,2.048 0.832,2.704 0.576,0.72 1.392,1.072 2.448,1.072 0.496,0 1.056,-0.128 1.712,-0.368 v -1.712 c -0.464,0.304 -1.008,0.464 -1.6,0.464 -0.944,0 -1.44,-0.512 -1.52,-1.536 z m -2.576,-2.672 c 0.64,0 0.96,0.4 0.976,1.216 h -1.968 c 0.048,-0.816 0.368,-1.216 0.992,-1.216 z"
-                    id="path24" />
-                    <use
-                    xlink:href="#path16"
-                    transform="translate(40.438)"
-                    id="use26" />
-                    <path
-                    d="m 110.187,-25.15 c -0.496,0 -0.992,0.208 -1.472,0.64 v -0.576 h -1.76 c 0.032,0.176 0.048,0.56 0.048,1.184 v 4.912 c 0,0.496 -0.016,0.832 -0.048,1.008 h 1.856 c 0,-0.064 -0.048,-0.64 -0.048,-1.008 v -3.936 c 0.368,-0.352 0.736,-0.528 1.088,-0.528 0.784,0 1.168,0.608 1.152,1.808 l -0.016,2.16 c -0.016,0.752 -0.032,1.264 -0.064,1.504 h 1.92 c -0.048,-0.256 -0.064,-0.752 -0.064,-1.472 v -2.192 c 0,-0.944 -0.192,-1.744 -0.592,-2.384 -0.464,-0.752 -1.136,-1.136 -2,-1.12 z"
-                    id="path28" />
-                    <path
-                    d="m 123.877,-17.982 c 0.144,0.016 0.256,0.016 0.336,0 0,-0.192 -0.064,-0.768 -0.064,-1.36 v -1.856 c 0,-0.56 0.016,-1.056 0.064,-1.52 h -1.952 c 0.032,0.704 0.048,1.12 0.032,1.248 0,1.28 -0.512,1.92 -1.552,1.92 -1.264,0 -1.904,-1.264 -1.904,-3.776 0,-2.48 0.784,-3.728 2.352,-3.728 0.752,0 1.472,0.288 2.16,0.88 v -1.824 c -0.608,-0.528 -1.328,-0.8 -2.16,-0.8 -2.896,0 -4.416,1.904 -4.416,5.424 0,3.696 1.328,5.552 3.968,5.552 0.592,0 1.12,-0.128 1.584,-0.368 0.368,-0.208 0.624,-0.432 0.768,-0.688 z"
-                    id="path30" />
-                    <path
-                    d="m 126.49,-26.334 c 0.592,0 1.104,-0.544 1.104,-1.184 0,-0.656 -0.512,-1.2 -1.104,-1.2 -0.624,0 -1.12,0.544 -1.12,1.2 0,0.64 0.496,1.184 1.12,1.184 z m 0.896,8.352 c -0.016,-0.24 -0.032,-0.64 -0.032,-1.184 v -4.912 c 0,-0.464 0.016,-0.8 0.032,-1.008 h -1.808 c 0.016,0.192 0.032,0.544 0.032,1.04 v 4.976 c 0,0.512 -0.016,0.88 -0.032,1.088 z"
-                    id="path32" />
-                    <path
-                    d="m 130.783,-25.742 c 0,-0.256 0.016,-0.48 0.048,-0.688 h -1.856 c 0.032,0.176 0.048,0.416 0.048,0.72 v 0.624 h -0.784 v 1.552 c 0.224,-0.032 0.4,-0.048 0.544,-0.048 l 0.24,0.016 v 0.032 h -0.016 v 2.864 c 0,0.896 0.112,1.552 0.336,1.968 0.304,0.56 0.832,0.832 1.616,0.832 0.56,0 1.024,-0.112 1.424,-0.32 v -1.6 c -0.272,0.176 -0.56,0.272 -0.896,0.272 -0.464,0 -0.704,-0.352 -0.704,-1.072 v -2.976 h 0.688 c 0.256,0 0.592,0.032 0.704,0.032 v -1.552 h -1.392 z"
-                    id="path34" />
-                    <path
-                    d="m 140.259,-27.678 c 0,-0.416 0.016,-0.736 0.064,-0.976 h -2.096 c 0.048,0.24 0.064,0.688 0.064,1.344 v 2.8 h -2.912 v -3.024 c 0,-0.48 0.016,-0.848 0.064,-1.12 h -2.08 c 0.048,0.256 0.064,0.624 0.064,1.12 v 8.432 c 0,0.496 -0.016,0.864 -0.064,1.12 h 2.08 c -0.048,-0.24 -0.064,-0.656 -0.064,-1.232 v -3.552 h 2.912 v 3.568 c 0,0.528 -0.016,0.944 -0.064,1.216 h 2.096 c -0.048,-0.24 -0.064,-0.624 -0.064,-1.12 v -3.664 h 0.528 v -1.744 h -0.528 z"
-                    id="path36" />
-                    <path
-                    d="m 144.402,-17.918 c 0.56,0 1.072,-0.208 1.568,-0.64 v 0.576 h 1.744 c -0.016,-0.176 -0.032,-0.576 -0.032,-1.2 v -4.896 c 0,-0.496 0.016,-0.832 0.032,-1.008 h -1.856 c 0,0.048 0.064,0.64 0.064,1.008 v 3.936 c -0.368,0.352 -0.704,0.528 -1.008,0.528 -0.432,0 -0.72,-0.16 -0.88,-0.496 -0.144,-0.272 -0.208,-0.704 -0.208,-1.312 l 0.016,-2.16 c 0.016,-0.768 0.032,-1.264 0.064,-1.504 h -1.92 c 0.048,0.256 0.064,0.752 0.064,1.472 v 2.192 c 0,0.976 0.16,1.776 0.48,2.384 0.4,0.752 1.024,1.12 1.872,1.12 z"
-                    id="path38" />
-                    <path
-                    d="m 152.31,-17.934 c 0.848,0 1.536,-0.416 2.048,-1.232 0.432,-0.704 0.64,-1.536 0.64,-2.48 0,-0.928 -0.208,-1.712 -0.608,-2.368 -0.48,-0.752 -1.136,-1.12 -1.984,-1.12 -0.464,0 -0.944,0.16 -1.44,0.464 v -2.784 c 0,-0.608 0.016,-1.008 0.032,-1.2 h -1.824 c 0.032,0.176 0.048,0.576 0.048,1.2 v 8.464 c 0,0.496 -0.016,0.832 -0.048,1.008 h 1.696 v -0.576 c 0.384,0.416 0.864,0.624 1.44,0.624 z m -0.24,-5.52 c 0.736,0 1.104,0.608 1.104,1.808 0,0.496 -0.08,0.928 -0.256,1.296 -0.208,0.464 -0.528,0.688 -0.944,0.688 -0.336,0 -0.672,-0.16 -1.008,-0.496 v -2.768 c 0.384,-0.352 0.752,-0.528 1.104,-0.528 z"
-                    id="path40" />
-                </g>
-                <path
-                    d="m 52,0 94,94 M 14,0 146,132"
-                    style="fill:none;stroke:#ffffff;stroke-dasharray:2, 1;stroke-opacity:0.95"
-                    id="path44" />
-                </svg>
-        </div>
-    </a>
-    <div class="container">
-		<div class="logo">
-			<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="250" height="250" viewBox="0 0 512 512">
-				<path fill="#7AA7D1" d="M262.074 485.246C254.809 485.265 247.407 485.534 240.165 484.99L226.178 483.306C119.737 468.826 34.1354 383.43 25.3176 274.714C24.3655 262.975 23.5876 253.161 24.3295 241.148C31.4284 126.212 123.985 31.919 238.633 24.1259L250.022 23.8366C258.02 23.8001 266.212 23.491 274.183 24.1306C320.519 27.8489 366.348 45.9743 402.232 75.4548L416.996 88.2751C444.342 114.373 464.257 146.819 475.911 182.72L480.415 197.211C486.174 219.054 488.67 242.773 487.436 265.259L486.416 275.75C478.783 352.041 436.405 418.1 369.36 455.394L355.463 462.875C326.247 477.031 294.517 484.631 262.074 485.246ZM253.547 72.4475C161.905 73.0454 83.5901 144.289 73.0095 234.5C69.9101 260.926 74.7763 292.594 83.9003 317.156C104.53 372.691 153.9 416.616 211.281 430.903C226.663 434.733 242.223 436.307 258.044 436.227C353.394 435.507 430.296 361.835 438.445 267.978C439.794 252.442 438.591 236.759 435.59 221.5C419.554 139.955 353.067 79.4187 269.856 72.7052C264.479 72.2714 258.981 72.423 253.586 72.4127L253.547 72.4475Z"/>
-				<path fill="#7AA7D1" d="M153.196 310.121L133.153 285.021C140.83 283.798 148.978 285.092 156.741 284.353L156.637 277.725L124.406 278.002C123.298 277.325 122.856 276.187 122.058 275.193L116.089 267.862C110.469 260.975 103.827 254.843 98.6026 247.669C103.918 246.839 105.248 246.537 111.14 246.523L129.093 246.327C130.152 238.785 128.62 240.843 122.138 240.758C111.929 240.623 110.659 242.014 105.004 234.661L97.9953 225.654C94.8172 221.729 91.2219 218.104 88.2631 214.005C84.1351 208.286 90.1658 209.504 94.601 209.489L236.752 209.545C257.761 209.569 268.184 211.009 285.766 221.678L285.835 206.051C285.837 197.542 286.201 189.141 284.549 180.748C280.22 158.757 260.541 143.877 240.897 135.739C238.055 134.561 232.259 133.654 235.575 129.851C244.784 119.288 263.68 111.99 277.085 111.105C288.697 109.828 301.096 113.537 311.75 117.703C360.649 136.827 393.225 183.042 398.561 234.866C402.204 270.253 391.733 308.356 367.999 335.1C332.832 374.727 269.877 384.883 223.294 360.397C206.156 351.388 183.673 333.299 175.08 316.6C173.511 313.551 174.005 313.555 170.443 313.52L160.641 313.449C158.957 313.435 156.263 314.031 155.122 312.487L153.196 310.121Z"/>
-			</svg>
-		</div>
-        <header>
-            <h1 class="center-align logo-title">ladddddddder</h1>
-        </header>
-        <form id="inputForm" class="col s12" method="get">
-            <div class="row">
-                <div class="input-field col s12">
-                    <input type="text" id="inputField" name="inputField" class="validate" required>
-                    <label for="inputField">URL</label>
-                </div>
-				<!--
-                <div class="input-field col s2">
-                    <button class="btn waves-effect waves-light" type="submit" name="action">Submit
-                        <i class="material-icons right">go</i>
-                    </button>
-                </div>
-				-->
-            </div>
-        </form>
-    </div>
-
-    <script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
-    <script>
-        document.addEventListener('DOMContentLoaded', function() {
-            M.AutoInit();
-        });
-        document.getElementById('inputForm').addEventListener('submit', function (e) {
-            e.preventDefault();
-            let url = document.getElementById('inputField').value;
-            if (url.indexOf('http') === -1) {
-                url = 'https://' + url;
-            }
-            window.location.href = '/' + url;
-            return false;
-        });
-    </script>
-</body>
-</html>
-`

handlers/proxy.go

@@ -18,6 +18,7 @@ import (
 var UserAgent = getenv("USER_AGENT", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
 var ForwardedFor = getenv("X_FORWARDED_FOR", "66.249.66.1")
 var rulesSet = loadRules()
+var allowedDomains = strings.Split(os.Getenv("ALLOWED_DOMAINS"), ",")
 
 func ProxySite(c *fiber.Ctx) error {
 	// Get the url from the URL
@@ -51,6 +52,10 @@ func fetchSite(urlpath string, queries map[string]string) (string, *http.Request
 		return "", nil, nil, err
 	}
 
+	if len(allowedDomains) > 0 && !StringInSlice(u.Host, allowedDomains) {
+		return "", nil, nil, fmt.Errorf("domain not allowed. %s not in %s", u.Host, allowedDomains)
+	}
+
 	if os.Getenv("DEBUG	") == "true" {
 		log.Println(u.String() + urlQuery)
 	}
@@ -98,7 +103,7 @@ func rewriteHtml(bodyB []byte, u *url.URL) string {
 	body = strings.ReplaceAll(body, "url(/", "url(/https://"+u.Host+"/")
 	body = strings.ReplaceAll(body, "href=\"https://"+u.Host, "href=\"/https://"+u.Host+"/")
 
-	if os.Getenv("RULES_URL") != "" {
+	if os.Getenv("RULESET") != "" {
 		body = applyRules(u.Host, u.Path, body)
 	}
 	return body
@@ -113,32 +118,48 @@ func getenv(key, fallback string) string {
 }
 
 func loadRules() RuleSet {
-	rulesUrl := os.Getenv("RULES_URL")
+	rulesUrl := os.Getenv("RULESET")
 	if rulesUrl == "" {
 		RulesList := RuleSet{}
 		return RulesList
 	}
 	log.Println("Loading rules")
 
-	resp, err := http.Get(rulesUrl)
-	if err != nil {
-		log.Println("ERROR:", err)
-	}
-	defer resp.Body.Close()
-
-	if resp.StatusCode >= 400 {
-		log.Println("ERROR:", resp.StatusCode, rulesUrl)
-	}
-
-	body, err := io.ReadAll(resp.Body)
-	if err != nil {
-		log.Println("ERROR:", err)
-	}
-
 	var ruleSet RuleSet
-	yaml.Unmarshal(body, &ruleSet)
+	if strings.HasPrefix(rulesUrl, "http") {
-	if err != nil {
+
-		log.Println("ERROR:", err)
+		resp, err := http.Get(rulesUrl)
+		if err != nil {
+			log.Println("ERROR:", err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode >= 400 {
+			log.Println("ERROR:", resp.StatusCode, rulesUrl)
+		}
+
+		body, err := io.ReadAll(resp.Body)
+		if err != nil {
+			log.Println("ERROR:", err)
+		}
+		yaml.Unmarshal(body, &ruleSet)
+
+		if err != nil {
+			log.Println("ERROR:", err)
+		}
+	} else {
+		yamlFile, err := os.ReadFile(rulesUrl)
+		if err != nil {
+			log.Println("ERROR:", err)
+		}
+		yaml.Unmarshal(yamlFile, &ruleSet)
+	}
+
+	for _, rule := range ruleSet {
+		//log.Println("Loaded rules for", rule.Domain)
+		if os.Getenv("ALLOWED_DOMAINS_RULESET") == "true" {
+			allowedDomains = append(allowedDomains, rule.Domain)
+		}
 	}
 
 	log.Println("Loaded rules for", len(ruleSet), "Domains")
@@ -154,7 +175,7 @@ func applyRules(domain string, path string, body string) string {
 		if rule.Domain != domain {
 			continue
 		}
-		if rule.Path != "" && rule.Path != path {
+		if len(rule.Paths) > 0 && !StringInSlice(path, rule.Paths) {
 			continue
 		}
 		for _, regexRule := range rule.RegexRules {
@@ -166,7 +187,15 @@ func applyRules(domain string, path string, body string) string {
 			if err != nil {
 				log.Fatal(err)
 			}
-			doc.Find(injection.Position).AppendHtml(injection.Code)
+			if injection.Replace != "" {
+				doc.Find(injection.Position).ReplaceWithHtml(injection.Replace)
+			}
+			if injection.Append != "" {
+				doc.Find(injection.Position).AppendHtml(injection.Append)
+			}
+			if injection.Prepend != "" {
+				doc.Find(injection.Position).PrependHtml(injection.Prepend)
+			}
 			body, err = doc.Html()
 			if err != nil {
 				log.Fatal(err)
@@ -183,12 +212,23 @@ type Rule struct {
 }
 
 type RuleSet []struct {
-	Domain      string `yaml:"domain"`
+	Domain      string   `yaml:"domain"`
-	Path        string `yaml:"path,omitempty"`
+	Paths       []string `yaml:"paths,omitempty"`
-	GoogleCache bool   `yaml:"googleCache,omitempty"`
+	GoogleCache bool     `yaml:"googleCache,omitempty"`
-	RegexRules  []Rule `yaml:"regexRules"`
+	RegexRules  []Rule   `yaml:"regexRules"`
 	Injections  []struct {
 		Position string `yaml:"position"`
-		Code     string `yaml:"code"`
+		Append   string `yaml:"append"`
+		Prepend  string `yaml:"prepend"`
+		Replace  string `yaml:"replace"`
 	} `yaml:"injections"`
 }
+
+func StringInSlice(s string, list []string) bool {
+	for _, x := range list {
+		if strings.HasPrefix(s, x) {
+			return true
+		}
+	}
+	return false
+}

handlers/ruleset.go

@@ -0,0 +1,24 @@
+package handlers
+
+import (
+	"os"
+
+	"github.com/gofiber/fiber/v2"
+	"gopkg.in/yaml.v3"
+)
+
+func Ruleset(c *fiber.Ctx) error {
+
+	if os.Getenv("EXPOSE_RULESET") == "false" {
+		c.SendStatus(fiber.StatusForbidden)
+		return c.SendString("Rules Disabled")
+	}
+
+	body, err := yaml.Marshal(rulesSet)
+	if err != nil {
+		c.SendStatus(fiber.StatusInternalServerError)
+		return c.SendString(err.Error())
+	}
+
+	return c.SendString(string(body))
+}

ruleset.yaml

@@ -4,22 +4,52 @@
       replace: <script $1 script="/https://www.example.com/$3"
   injections:
     - position: head # Position where to inject the code
-      code: | 
+      append: | 
         <script>
           window.localStorage.clear();
           console.log("test");
           alert("Hello!");
         </script>
-- domain: www.anotherdomain.com # Domain where the rule applies
+    - position: h1
-  path: /article                # Path where the rule applies
+      replace: | 
-  googleCache: false            # Search also in Google Cache
+        <h1>An example with a ladder ;-)</h1>
-  regexRules:                   # Regex rules to apply
+- domain: www.americanbanker.com
-    - match: <script\s+([^>]*\s+)?src="(/)([^"]*)"
+  paths: 
-      replace: <script $1 script="/https://www.example.com/$3"
+    - /news
   injections:
-    - position: .left-content article .post-title # Position where to inject the code into DOM
+    - position: head
-      code: | 
+      append: | 
         <script>
-          window.localStorage.clear();
+          document.addEventListener("DOMContentLoaded", () => {
-          console.log("test");
+            const inlineGate = document.querySelector('.inline-gate');
+            if (inlineGate) {
+              inlineGate.classList.remove('inline-gate');
+              const inlineGated = document.querySelectorAll('.inline-gated');
+              for (const elem of inlineGated) { elem.classList.remove('inline-gated'); }
+            }
+          });
+        </script>
+- domain: www.nzz.ch
+  paths: 
+    - /international
+    - /sport
+    - /wirtschaft
+    - /technologie
+    - /feuilleton
+    - /zuerich
+    - /wissenschaft
+    - /gesellschaft
+    - /panorama
+    - /mobilitaet
+    - /reisen
+    - /meinung
+    - /finanze
+  injections:
+    - position: head
+      append: | 
+        <script>
+          document.addEventListener("DOMContentLoaded", () => {
+            const paywall = document.querySelector('.dynamic-regwall');
+            removeDOMElement(paywall)
+          });
         </script>