Compare commits
28 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
890d6929e0 | ||
|
|
8a6320ccca | ||
|
|
67f9af0296 | ||
|
|
cde6ed7229 | ||
|
|
96dd4de876 | ||
|
|
d5c58f42da | ||
|
|
d34c5680b3 | ||
|
|
6dfdaaa25b | ||
|
|
7ea7f253e8 | ||
|
|
bf2529753d | ||
|
|
e3389d2df3 | ||
|
|
b786796595 | ||
|
|
377a577c67 | ||
|
|
6eb7b481d8 | ||
|
|
2f1de95e06 | ||
|
|
7ae1a29932 | ||
|
|
63dcaeba3c | ||
|
|
62e03a384a | ||
|
|
7f4d749c55 | ||
|
|
e748cb09a5 | ||
|
|
c98e49f2b3 | ||
|
|
3e3eebcdc2 | ||
|
|
45a3fe2adf | ||
|
|
ec7f2089fc | ||
|
|
622143211c | ||
|
|
796ffd1c87 | ||
|
|
210837a404 | ||
|
|
aa1caaca98 |
2
.github/workflows/release-binaries.yaml
vendored
2
.github/workflows/release-binaries.yaml
vendored
@@ -22,7 +22,7 @@ jobs:
|
||||
-
|
||||
name: Set version
|
||||
run: |
|
||||
echo -n $(git describe --tags --abbrev=0) > cmd/VERSION
|
||||
echo -n $(git describe --tags --abbrev=0) > handlers/VERSION
|
||||
-
|
||||
name: Set up Go
|
||||
uses: actions/setup-go@v3
|
||||
|
||||
2
.github/workflows/release-docker.yaml
vendored
2
.github/workflows/release-docker.yaml
vendored
@@ -42,7 +42,7 @@ jobs:
|
||||
- name: Set version
|
||||
id: version
|
||||
run: |
|
||||
echo ${GITHUB_REF#refs/tags/v} > cmd/VERSION
|
||||
echo ${GITHUB_REF#refs/tags/v} > handlers/VERSION
|
||||
|
||||
# Install the cosign tool except on PR
|
||||
# https://github.com/sigstore/cosign-installer
|
||||
|
||||
@@ -7,7 +7,7 @@ before:
|
||||
builds:
|
||||
-
|
||||
main: cmd/main.go
|
||||
binary: kubero
|
||||
binary: ladder
|
||||
env:
|
||||
- CGO_ENABLED=0
|
||||
goos:
|
||||
|
||||
67
README.md
67
README.md
@@ -1,5 +1,5 @@
|
||||
<p align="center">
|
||||
<img src="public/pigeon.svg" width="100px">
|
||||
<img src="assets/pigeon.svg" width="100px">
|
||||
</p>
|
||||
|
||||
<h1 align="center">Ladder</h1>
|
||||
@@ -10,22 +10,35 @@
|
||||
|
||||
Freedom of information is an essential pillar of democracy and informed decision-making. While media organizations have legitimate financial interests, it is crucial to strike a balance between profitability and the public's right to access information. The proliferation of paywalls raises concerns about the erosion of this fundamental freedom, and it is imperative for society to find innovative ways to preserve access to vital information without compromising the sustainability of journalism. In a world where knowledge should be shared and not commodified, paywalls should be critically examined to ensure that they do not undermine the principles of an open and informed society.
|
||||
|
||||
Some site might have missing images or other formating issues. This is due to the fact that the site using javascript or CSS to load the images/JS/CSS. This is a limitation of this proxy. If you prefer a full experience, please concider buying a subscription for the site.
|
||||
Certain sites may display missing images or encounter formatting issues. This can be attributed to the site's reliance on JavaScript or CSS for image and resource loading, which presents a limitation when accessed through this proxy. If you prefer a full experience, please concider buying a subscription for the site.
|
||||
|
||||
> **Disclaimer:** This project is intended for educational purposes only. The author does not endorse or encourage any unethical or illegal activity. Use this tool at your own risk.
|
||||
|
||||
### Features
|
||||
- [x] Bypass Paywalls
|
||||
- [x] Remove CORS
|
||||
- [x] Keep Site browsable
|
||||
- [x] Docker Container
|
||||
- [x] Linux Binary
|
||||
- [x] Mac OS Binary
|
||||
- [x] Windows Binary (Untested)
|
||||
- [x] Remove CORS headers from responses, assets, and images ...
|
||||
- [x] Apply domain based ruleset/code to modify response
|
||||
- [x] Keep site browsable
|
||||
- [x] API
|
||||
- [x] Fetch RAW HTML
|
||||
- [x] Custom User Agent
|
||||
- [x] Custom X-Forwarded-For IP
|
||||
- [x] [Docker container](https://github.com/kubero-dev/ladder/pkgs/container/ladder)
|
||||
- [x] Linux binary
|
||||
- [x] Mac OS binary
|
||||
- [x] Windows binary (untested)
|
||||
- [x] Removes most of the ads (unexpected side effect)
|
||||
- [x] Basic Auth
|
||||
- [x] Disable logs
|
||||
- [x] No Tracking
|
||||
|
||||
## How to use
|
||||
## Installation
|
||||
|
||||
> **Warning:** If your instance will be publicly accessible, make sure to enable Basic Auth. This will prevent unauthorized users from using your proxy. If you do not enable Basic Auth, anyone can use your proxy to browse nasty/illegal stuff. And you will be responsible for it.
|
||||
|
||||
### Binary
|
||||
1) Download Binary
|
||||
2) Run Binary
|
||||
1) Download binary [here](https://github.com/kubero-dev/ladder/releases/latest)
|
||||
2) Unpack and run the binary `./ladder`
|
||||
3) Open Browser (Default: http://localhost:8080)
|
||||
|
||||
### Docker
|
||||
@@ -35,10 +48,28 @@ docker run -p 8080:8080 -d --name ladder ghcr.io/kubero-dev/ladder:latest
|
||||
|
||||
### Docker Compose
|
||||
```bash
|
||||
wget https://raw.githubusercontent.com/kubero-dev/ladder/main/docker-compose.yml
|
||||
curl https://raw.githubusercontent.com/kubero-dev/ladder/main/docker-compose.yaml --output docker-compose.yaml
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Browser
|
||||
1) Open Browser (Default: http://localhost:8080)
|
||||
2) Enter URL
|
||||
3) Press Enter
|
||||
|
||||
Or direct by appending the URL to the end of the proxy URL:
|
||||
http://localhost:8080/https://www.example.com
|
||||
|
||||
### API
|
||||
```bash
|
||||
curl -X GET "http://localhost:8080/api/https://www.example.com"
|
||||
```
|
||||
|
||||
### RAW
|
||||
http://localhost:8080/raw/https://www.example.com
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
@@ -47,3 +78,15 @@ docker-compose up -d
|
||||
| --- | --- | --- |
|
||||
| `PORT` | Port to listen on | `8080` |
|
||||
| `PREFORK` | Spawn multiple server instances | `false` |
|
||||
| `USER_AGENT` | User agent to emulate | `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` |
|
||||
| `X_FORWARDED_FOR` | IP forwarder address | `66.249.66.1` |
|
||||
| `USERPASS` | Enables Basic Auth, format `admin:123456` | `` |
|
||||
| `LOG_URLS` | Log fetched URL's | `true` |
|
||||
| `DISABLE_FORM` | Disables URL Form Frontpage | `false` |
|
||||
| `RULES_URL` | URL to a ruleset file | `https://raw.githubusercontent.com/kubero-dev/ladder/main/ruleset.yaml` |
|
||||
|
||||
### Ruleset
|
||||
|
||||
It is possible to apply custom rules to modify the response. This can be used to remove unwanted or modify elements from the page. The ruleset is a YAML file that contains a list of rules for each domain and is loaded on startup
|
||||
|
||||
See in [ruleset.yaml](ruleset.yaml) for an example.
|
||||
@@ -159,8 +159,11 @@
|
||||
});
|
||||
document.getElementById('inputForm').addEventListener('submit', function (e) {
|
||||
e.preventDefault();
|
||||
const inputValue = document.getElementById('inputField').value;
|
||||
window.location.href = '/' + inputValue;
|
||||
let url = document.getElementById('inputField').value;
|
||||
if (url.indexOf('http') === -1) {
|
||||
url = 'https://' + url;
|
||||
}
|
||||
window.location.href = '/' + url;
|
||||
return false;
|
||||
});
|
||||
</script>
|
||||
|
Before Width: | Height: | Size: 2.3 KiB After Width: | Height: | Size: 2.3 KiB |
BIN
cmd/favicon.ico
Normal file
BIN
cmd/favicon.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
34
cmd/main.go
34
cmd/main.go
@@ -1,14 +1,22 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
|
||||
"ladder/handlers"
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
"github.com/gofiber/fiber/v2/middleware/basicauth"
|
||||
"github.com/gofiber/fiber/v2/middleware/favicon"
|
||||
)
|
||||
|
||||
//go:embed favicon.ico
|
||||
var faviconData string
|
||||
|
||||
func main() {
|
||||
|
||||
prefork, _ := strconv.ParseBool(os.Getenv("PREFORK"))
|
||||
@@ -18,8 +26,32 @@ func main() {
|
||||
},
|
||||
)
|
||||
|
||||
userpass := os.Getenv("USERPASS")
|
||||
if userpass != "" {
|
||||
userpass := strings.Split(userpass, ":")
|
||||
app.Use(basicauth.New(basicauth.Config{
|
||||
Users: map[string]string{
|
||||
userpass[0]: userpass[1],
|
||||
},
|
||||
}))
|
||||
}
|
||||
|
||||
app.Use(favicon.New(favicon.Config{
|
||||
Data: []byte(faviconData),
|
||||
URL: "/favicon.ico",
|
||||
}))
|
||||
|
||||
if os.Getenv("NOLOGS") != "true" {
|
||||
app.Use(func(c *fiber.Ctx) error {
|
||||
log.Println(c.Method(), c.Path())
|
||||
return c.Next()
|
||||
})
|
||||
}
|
||||
|
||||
app.Get("/", handlers.Form)
|
||||
app.Get("debug/*", handlers.Debug)
|
||||
|
||||
app.Get("raw/*", handlers.Raw)
|
||||
app.Get("api/*", handlers.Api)
|
||||
app.Get("/*", handlers.ProxySite)
|
||||
|
||||
port := os.Getenv("PORT")
|
||||
|
||||
@@ -5,12 +5,15 @@ services:
|
||||
container_name: ladder
|
||||
build: .
|
||||
#restart: always
|
||||
#command: tail -f /dev/null
|
||||
#command: sh -c ./ladder
|
||||
environment:
|
||||
- PORT=8080
|
||||
- PREFORK=true
|
||||
#- GODEBUG=netdns=go+4
|
||||
#- PREFORK=true
|
||||
#- X_FORWARDED_FOR=66.249.66.1
|
||||
#- USER_AGENT=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
||||
#- USERPASS=foo:bar
|
||||
#- LOG_URLS=true
|
||||
#- GODEBUG=netdns=go
|
||||
ports:
|
||||
- "8080:8080"
|
||||
deploy:
|
||||
|
||||
14
go.mod
14
go.mod
@@ -2,17 +2,29 @@ module ladder
|
||||
|
||||
go 1.21.1
|
||||
|
||||
require github.com/gofiber/fiber/v2 v2.50.0
|
||||
|
||||
require (
|
||||
github.com/andybalholm/cascadia v1.3.1 // indirect
|
||||
golang.org/x/net v0.8.0 // indirect
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/PuerkitoBio/goquery v1.8.1
|
||||
github.com/andybalholm/brotli v1.0.6 // indirect
|
||||
github.com/gofiber/fiber/v2 v2.50.0
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/google/uuid v1.4.0 // indirect
|
||||
github.com/klauspost/compress v1.17.2 // indirect
|
||||
github.com/mattn/go-colorable v0.1.13 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/mattn/go-runewidth v0.0.15 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/rivo/uniseg v0.4.4 // indirect
|
||||
github.com/stretchr/objx v0.5.0 // indirect
|
||||
github.com/stretchr/testify v1.8.4 // indirect
|
||||
github.com/valyala/bytebufferpool v1.0.0 // indirect
|
||||
github.com/valyala/fasthttp v1.50.0 // indirect
|
||||
github.com/valyala/tcplisten v1.0.0 // indirect
|
||||
golang.org/x/sys v0.13.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
53
go.sum
53
go.sum
@@ -1,5 +1,12 @@
|
||||
github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
|
||||
github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
|
||||
github.com/andybalholm/brotli v1.0.6 h1:Yf9fFpf49Zrxb9NlQaluyE92/+X7UVHlhMNJN2sxfOI=
|
||||
github.com/andybalholm/brotli v1.0.6/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
|
||||
github.com/andybalholm/cascadia v1.3.1 h1:nhxRkql1kdYCc8Snf7D5/D3spOX+dBgjA6u8x004T2c=
|
||||
github.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/gofiber/fiber/v2 v2.50.0 h1:ia0JaB+uw3GpNSCR5nvC5dsaxXjRU5OEu36aytx+zGw=
|
||||
github.com/gofiber/fiber/v2 v2.50.0/go.mod h1:21eytvay9Is7S6z+OgPi7c7n4++tnClWmhpimVHMimw=
|
||||
github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
|
||||
@@ -13,16 +20,62 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U=
|
||||
github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
|
||||
github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis=
|
||||
github.com/rivo/uniseg v0.4.4/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
|
||||
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
|
||||
github.com/valyala/fasthttp v1.50.0 h1:H7fweIlBm0rXLs2q0XbalvJ6r0CUPFWK3/bB4N13e9M=
|
||||
github.com/valyala/fasthttp v1.50.0/go.mod h1:k2zXd82h/7UZc3VOdJ2WaUqt1uZ/XpXAfE9i+HBC3lA=
|
||||
github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8=
|
||||
github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
||||
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||
golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
|
||||
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
|
||||
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
58
handlers/api.go
Normal file
58
handlers/api.go
Normal file
@@ -0,0 +1,58 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
|
||||
"log"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
)
|
||||
|
||||
//go:embed VERSION
|
||||
var version string
|
||||
|
||||
func Api(c *fiber.Ctx) error {
|
||||
// Get the url from the URL
|
||||
urlQuery := c.Params("*")
|
||||
|
||||
queries := c.Queries()
|
||||
body, req, resp, err := fetchSite(urlQuery, queries)
|
||||
if err != nil {
|
||||
log.Println("ERROR:", err)
|
||||
c.SendStatus(500)
|
||||
return c.SendString(err.Error())
|
||||
}
|
||||
|
||||
response := Response{
|
||||
Version: version,
|
||||
Body: body,
|
||||
}
|
||||
response.Request.Headers = make([]interface{}, 0)
|
||||
for k, v := range req.Header {
|
||||
response.Request.Headers = append(response.Request.Headers, map[string]string{
|
||||
"key": k,
|
||||
"value": v[0],
|
||||
})
|
||||
}
|
||||
|
||||
response.Response.Headers = make([]interface{}, 0)
|
||||
for k, v := range resp.Header {
|
||||
response.Response.Headers = append(response.Response.Headers, map[string]string{
|
||||
"key": k,
|
||||
"value": v[0],
|
||||
})
|
||||
}
|
||||
|
||||
return c.JSON(response)
|
||||
}
|
||||
|
||||
type Response struct {
|
||||
Version string `json:"version"`
|
||||
Body string `json:"body"`
|
||||
Request struct {
|
||||
Headers []interface{} `json:"headers"`
|
||||
} `json:"request"`
|
||||
Response struct {
|
||||
Headers []interface{} `json:"headers"`
|
||||
} `json:"response"`
|
||||
}
|
||||
44
handlers/api.test.go
Normal file
44
handlers/api.test.go
Normal file
@@ -0,0 +1,44 @@
|
||||
// BEGIN: 7d5e1f7c7d5e
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestApi(t *testing.T) {
|
||||
app := fiber.New()
|
||||
app.Get("/api/*", Api)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
url string
|
||||
expectedStatus int
|
||||
}{
|
||||
{
|
||||
name: "valid url",
|
||||
url: "https://www.google.com",
|
||||
expectedStatus: http.StatusOK,
|
||||
},
|
||||
{
|
||||
name: "invalid url",
|
||||
url: "invalid-url",
|
||||
expectedStatus: http.StatusBadRequest,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/"+tt.url, nil)
|
||||
resp, err := app.Test(req)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, tt.expectedStatus, resp.StatusCode)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// END: 7d5e1f7c7d5e
|
||||
@@ -1,44 +0,0 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
)
|
||||
|
||||
func Debug(c *fiber.Ctx) error {
|
||||
// Get the url from the URL
|
||||
urlQuery := c.Params("*")
|
||||
|
||||
u, err := url.Parse(urlQuery)
|
||||
if err != nil {
|
||||
return c.SendString(err.Error())
|
||||
}
|
||||
|
||||
log.Println(u.String())
|
||||
|
||||
// Fetch the site
|
||||
client := &http.Client{}
|
||||
req, _ := http.NewRequest("GET", u.String(), nil)
|
||||
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
|
||||
req.Header.Set("X-Forwarded-For", "66.249.66.1")
|
||||
req.Header.Set("Referer", u.String())
|
||||
req.Header.Set("Host", u.Host)
|
||||
resp, err := client.Do(req)
|
||||
|
||||
if err != nil {
|
||||
return c.SendString(err.Error())
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
bodyB, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
log.Println("ERROR", err)
|
||||
return c.SendString(err.Error())
|
||||
}
|
||||
body := rewriteHtml(bodyB, u)
|
||||
return c.SendString(body)
|
||||
}
|
||||
@@ -1,11 +1,21 @@
|
||||
package handlers
|
||||
|
||||
import "github.com/gofiber/fiber/v2"
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
)
|
||||
|
||||
func Form(c *fiber.Ctx) error {
|
||||
if os.Getenv("DISABLE_FORM") == "true" {
|
||||
c.Set("Content-Type", "text/html")
|
||||
c.SendStatus(fiber.StatusNotFound)
|
||||
return c.SendString("Form Disabled")
|
||||
} else {
|
||||
c.Set("Content-Type", "text/html")
|
||||
return c.SendString(html)
|
||||
}
|
||||
}
|
||||
|
||||
const html = `
|
||||
<!DOCTYPE html>
|
||||
@@ -169,8 +179,11 @@ const html = `
|
||||
});
|
||||
document.getElementById('inputForm').addEventListener('submit', function (e) {
|
||||
e.preventDefault();
|
||||
const inputValue = document.getElementById('inputField').value;
|
||||
window.location.href = '/' + inputValue;
|
||||
let url = document.getElementById('inputField').value;
|
||||
if (url.indexOf('http') === -1) {
|
||||
url = 'https://' + url;
|
||||
}
|
||||
window.location.href = '/' + url;
|
||||
return false;
|
||||
});
|
||||
</script>
|
||||
|
||||
@@ -6,49 +6,76 @@ import (
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/PuerkitoBio/goquery"
|
||||
"github.com/gofiber/fiber/v2"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
var UserAgent = getenv("USER_AGENT", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
|
||||
var ForwardedFor = getenv("X_FORWARDED_FOR", "66.249.66.1")
|
||||
var rulesSet = loadRules()
|
||||
|
||||
func ProxySite(c *fiber.Ctx) error {
|
||||
// Get the url from the URL
|
||||
urlQuery := c.Params("*")
|
||||
url := c.Params("*")
|
||||
|
||||
u, err := url.Parse(urlQuery)
|
||||
queries := c.Queries()
|
||||
body, _, resp, err := fetchSite(url, queries)
|
||||
if err != nil {
|
||||
log.Println("ERROR", err)
|
||||
c.SendStatus(500)
|
||||
log.Println("ERROR:", err)
|
||||
c.SendStatus(fiber.StatusInternalServerError)
|
||||
return c.SendString(err.Error())
|
||||
}
|
||||
|
||||
log.Println(u.String())
|
||||
c.Set("Content-Type", resp.Header.Get("Content-Type"))
|
||||
return c.SendString(body)
|
||||
}
|
||||
|
||||
func fetchSite(urlpath string, queries map[string]string) (string, *http.Request, *http.Response, error) {
|
||||
|
||||
urlQuery := "?"
|
||||
if len(queries) > 0 {
|
||||
for k, v := range queries {
|
||||
urlQuery += k + "=" + v + "&"
|
||||
}
|
||||
}
|
||||
urlQuery = strings.TrimSuffix(urlQuery, "&")
|
||||
urlQuery = strings.TrimSuffix(urlQuery, "?")
|
||||
|
||||
u, err := url.Parse(urlpath)
|
||||
if err != nil {
|
||||
return "", nil, nil, err
|
||||
}
|
||||
|
||||
if os.Getenv("DEBUG ") == "true" {
|
||||
log.Println(u.String() + urlQuery)
|
||||
}
|
||||
|
||||
// Fetch the site
|
||||
client := &http.Client{}
|
||||
req, _ := http.NewRequest("GET", u.String(), nil)
|
||||
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
|
||||
req.Header.Set("X-Forwarded-For", "66.249.66.1")
|
||||
req, _ := http.NewRequest("GET", u.String()+urlQuery, nil)
|
||||
req.Header.Set("User-Agent", UserAgent)
|
||||
req.Header.Set("X-Forwarded-For", ForwardedFor)
|
||||
req.Header.Set("Referer", u.String())
|
||||
req.Header.Set("Host", u.Host)
|
||||
resp, err := client.Do(req)
|
||||
|
||||
if err != nil {
|
||||
return c.SendString(err.Error())
|
||||
return "", nil, nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
bodyB, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
log.Println("ERROR", err)
|
||||
c.SendStatus(500)
|
||||
return c.SendString(err.Error())
|
||||
return "", nil, nil, err
|
||||
}
|
||||
|
||||
body := rewriteHtml(bodyB, u)
|
||||
c.Set("Content-Type", resp.Header.Get("Content-Type"))
|
||||
return c.SendString(body)
|
||||
return body, req, resp, nil
|
||||
}
|
||||
|
||||
func rewriteHtml(bodyB []byte, u *url.URL) string {
|
||||
@@ -63,7 +90,7 @@ func rewriteHtml(bodyB []byte, u *url.URL) string {
|
||||
// scripts
|
||||
scriptPattern := `<script\s+([^>]*\s+)?src="(/)([^"]*)"`
|
||||
reScript := regexp.MustCompile(scriptPattern)
|
||||
body = reScript.ReplaceAllString(body, fmt.Sprintf(`<script $1 script="%s$3"`, "/https://"+u.Host))
|
||||
body = reScript.ReplaceAllString(body, fmt.Sprintf(`<script $1 script="%s$3"`, "/https://"+u.Host+"/"))
|
||||
|
||||
//body = strings.ReplaceAll(body, "srcset=\"/", "srcset=\"/https://"+u.Host+"/") // TODO: Needs a regex to rewrite the URL's
|
||||
body = strings.ReplaceAll(body, "href=\"/", "href=\"/https://"+u.Host+"/")
|
||||
@@ -71,5 +98,97 @@ func rewriteHtml(bodyB []byte, u *url.URL) string {
|
||||
body = strings.ReplaceAll(body, "url(/", "url(/https://"+u.Host+"/")
|
||||
body = strings.ReplaceAll(body, "href=\"https://"+u.Host, "href=\"/https://"+u.Host+"/")
|
||||
|
||||
if os.Getenv("RULES_URL") != "" {
|
||||
body = applyRules(u.Host, u.Path, body)
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
value := os.Getenv(key)
|
||||
if len(value) == 0 {
|
||||
return fallback
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func loadRules() RuleSet {
|
||||
rulesUrl := os.Getenv("RULES_URL")
|
||||
if rulesUrl == "" {
|
||||
RulesList := RuleSet{}
|
||||
return RulesList
|
||||
}
|
||||
log.Println("Loading rules")
|
||||
|
||||
resp, err := http.Get(rulesUrl)
|
||||
if err != nil {
|
||||
log.Println("ERROR:", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode >= 400 {
|
||||
log.Println("ERROR:", resp.StatusCode, rulesUrl)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
log.Println("ERROR:", err)
|
||||
}
|
||||
|
||||
var ruleSet RuleSet
|
||||
yaml.Unmarshal(body, &ruleSet)
|
||||
if err != nil {
|
||||
log.Println("ERROR:", err)
|
||||
}
|
||||
|
||||
log.Println("Loaded rules for", len(ruleSet), "Domains")
|
||||
return ruleSet
|
||||
}
|
||||
|
||||
func applyRules(domain string, path string, body string) string {
|
||||
if len(rulesSet) == 0 {
|
||||
return body
|
||||
}
|
||||
|
||||
for _, rule := range rulesSet {
|
||||
if rule.Domain != domain {
|
||||
continue
|
||||
}
|
||||
if rule.Path != "" && rule.Path != path {
|
||||
continue
|
||||
}
|
||||
for _, regexRule := range rule.RegexRules {
|
||||
re := regexp.MustCompile(regexRule.Match)
|
||||
body = re.ReplaceAllString(body, regexRule.Replace)
|
||||
}
|
||||
for _, injection := range rule.Injections {
|
||||
doc, err := goquery.NewDocumentFromReader(strings.NewReader(body))
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
doc.Find(injection.Position).AppendHtml(injection.Code)
|
||||
body, err = doc.Html()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return body
|
||||
}
|
||||
|
||||
type Rule struct {
|
||||
Match string `yaml:"match"`
|
||||
Replace string `yaml:"replace"`
|
||||
}
|
||||
|
||||
type RuleSet []struct {
|
||||
Domain string `yaml:"domain"`
|
||||
Path string `yaml:"path,omitempty"`
|
||||
GoogleCache bool `yaml:"googleCache,omitempty"`
|
||||
RegexRules []Rule `yaml:"regexRules"`
|
||||
Injections []struct {
|
||||
Position string `yaml:"position"`
|
||||
Code string `yaml:"code"`
|
||||
} `yaml:"injections"`
|
||||
}
|
||||
|
||||
58
handlers/proxy.test.go
Normal file
58
handlers/proxy.test.go
Normal file
@@ -0,0 +1,58 @@
|
||||
// BEGIN: 6f8b3f5d5d5d
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestProxySite(t *testing.T) {
|
||||
app := fiber.New()
|
||||
app.Get("/:url", ProxySite)
|
||||
|
||||
req := httptest.NewRequest("GET", "/https://example.com", nil)
|
||||
resp, err := app.Test(req)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, http.StatusOK, resp.StatusCode)
|
||||
}
|
||||
|
||||
func TestRewriteHtml(t *testing.T) {
|
||||
bodyB := []byte(`
|
||||
<html>
|
||||
<head>
|
||||
<title>Test Page</title>
|
||||
</head>
|
||||
<body>
|
||||
<img src="/image.jpg">
|
||||
<script src="/script.js"></script>
|
||||
<a href="/about">About Us</a>
|
||||
<div style="background-image: url('/background.jpg')"></div>
|
||||
</body>
|
||||
</html>
|
||||
`)
|
||||
u := &url.URL{Host: "example.com"}
|
||||
|
||||
expected := `
|
||||
<html>
|
||||
<head>
|
||||
<title>Test Page</title>
|
||||
</head>
|
||||
<body>
|
||||
<img src="/https://example.com/image.jpg">
|
||||
<script script="/https://example.com/script.js"></script>
|
||||
<a href="/https://example.com/about">About Us</a>
|
||||
<div style="background-image: url('/https://example.com/background.jpg')"></div>
|
||||
</body>
|
||||
</html>
|
||||
`
|
||||
|
||||
actual := rewriteHtml(bodyB, u)
|
||||
assert.Equal(t, expected, actual)
|
||||
}
|
||||
|
||||
// END: 6f8b3f5d5d5d
|
||||
21
handlers/raw.go
Normal file
21
handlers/raw.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"log"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
)
|
||||
|
||||
func Raw(c *fiber.Ctx) error {
|
||||
// Get the url from the URL
|
||||
urlQuery := c.Params("*")
|
||||
|
||||
queries := c.Queries()
|
||||
body, _, _, err := fetchSite(urlQuery, queries)
|
||||
if err != nil {
|
||||
log.Println("ERROR:", err)
|
||||
c.SendStatus(500)
|
||||
return c.SendString(err.Error())
|
||||
}
|
||||
return c.SendString(body)
|
||||
}
|
||||
60
handlers/raw.test.go
Normal file
60
handlers/raw.test.go
Normal file
@@ -0,0 +1,60 @@
|
||||
// BEGIN: 7f8d9e6d4b5c
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gofiber/fiber/v2"
|
||||
)
|
||||
|
||||
func TestRaw(t *testing.T) {
|
||||
app := fiber.New()
|
||||
app.Get("/raw/*", Raw)
|
||||
|
||||
testCases := []struct {
|
||||
name string
|
||||
url string
|
||||
expected string
|
||||
}{
|
||||
{
|
||||
name: "valid url",
|
||||
url: "https://www.google.com",
|
||||
expected: "<!doctype html>",
|
||||
},
|
||||
{
|
||||
name: "invalid url",
|
||||
url: "invalid-url",
|
||||
expected: "parse invalid-url: invalid URI for request",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range testCases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, "/raw/"+tc.url, nil)
|
||||
resp, err := app.Test(req)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Errorf("expected status OK; got %v", resp.Status)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if !strings.Contains(string(body), tc.expected) {
|
||||
t.Errorf("expected body to contain %q; got %q", tc.expected, string(body))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// END: 7f8d9e6d4b5c
|
||||
25
ruleset.yaml
Normal file
25
ruleset.yaml
Normal file
@@ -0,0 +1,25 @@
|
||||
- domain: www.example.com
|
||||
regexRules:
|
||||
- match: <script\s+([^>]*\s+)?src="(/)([^"]*)"
|
||||
replace: <script $1 script="/https://www.example.com/$3"
|
||||
injections:
|
||||
- position: head # Position where to inject the code
|
||||
code: |
|
||||
<script>
|
||||
window.localStorage.clear();
|
||||
console.log("test");
|
||||
alert("Hello!");
|
||||
</script>
|
||||
- domain: www.anotherdomain.com # Domain where the rule applies
|
||||
path: /article # Path where the rule applies
|
||||
googleCache: false # Search also in Google Cache
|
||||
regexRules: # Regex rules to apply
|
||||
- match: <script\s+([^>]*\s+)?src="(/)([^"]*)"
|
||||
replace: <script $1 script="/https://www.example.com/$3"
|
||||
injections:
|
||||
- position: .left-content article .post-title # Position where to inject the code into DOM
|
||||
code: |
|
||||
<script>
|
||||
window.localStorage.clear();
|
||||
console.log("test");
|
||||
</script>
|
||||
Reference in New Issue
Block a user