sync

docs/drafts/20100611_3_url_shortener.txt

@@ -0,0 +1,105 @@
+URL shortener service (to review / full SPEC / DR)
+
+URL shortener service aims to provide short encoded URLs on demand based on given output. it aims to be a helper to some other service and should act as a part of post-alpha (next to alpha) stages, but the implementation priority may be reordered depending on:
+- the BE capacity
+- the BE load
+- the product urgency
+
+Basic requirements:
+- URL shortener shouldn't be a part of the BE application - the development should be separated out of it and act like a standalone opt-in service
+- should include REST API (Open API) - as a must
+- stack: PHP, NGINX, MariaDB, any free / open-source licensed 3rd parties if needed
+- the placement inside the project structure: backend/url-shortener submodule
+- the development placement: git://f0xx.org/androidcast_project/url-shortener (will be available soon)
+- QR codes support: must
+- authentication: app-wide (shared session), project-wise token
+- permalink urls: must
+- time-restricted urls: must
+- token signing, trust models, sessions, security, other options: TBD or described later on
+- ... TBD - open part
+
+The high-level flow:
+- some app (not requirely androidcast - for example, curl) requests url shortener to shorten the long url
+- some app provides the authentication on the way we delegate to it (temporary time-restricted or persistent)
+- url shortener generates JSON with response of the operation
+
+The request:
+1) GET: https://s.f0xx.org?url=[URL]&bearer=[TOKEN]&signer=[SIGNER]&ttl=TTL
+URL - url-encoded string to shorten
+TOKEN - app-wise, demo or personal token to attach the shortened url to into the database
+SIGNER - trusted signer token
+TTL - time-to-live for shortened URL
+2) POST: JSON-based body with the same fields as for 1) GET
+
+The response:
+1) fail: {"code": "ERRORCODE", "url": "URL", "bearer": "TOKEN", "desc": "ERROR_DESCRIPTION"}
+ERRORCODE - internal error code
+URL - the URL to shorten
+TOKEN - auth token
+DESC - error description (for example "access denied")
+2) success: {"code": "0", "url": "ORIGINAL_URL", "u": "SHORTENED_URL", "bearer": "TOKEN", "ttl": "TTL"}
+ORIGINAL_URL - the original URL
+SHORTENED_URL - short URL
+TOKEN - auth token
+TTL - time to live for this SHORTENED_URL or 0 if forever
+
+The low-level flow:
+1) web app receives the Request to shorten url
+2) web app checks if the TOKEN is already active (may be demo token or pre-shared special)
+3) if TOKEN is absent - return the error response
+4) if the TOKEN persists, do a lookup for the URL (already shortened) through the database that matches this TOKEN, return cached one if exists and TTL is good
+5) if shortened URL found but TTL exceeds given limits - return error response
+6) if not exists, check if the TOKEN is in whitelist (trusted)
+7) if not trusted - return error response
+8) if trusted token and the shortened URL exists and TTL is ok - return Ok response with corrected TTL in the response body (remaining)
+9) if trusted token and URL doesn't exist in the database,
+9.1) check if the TOKEN is really trusted to
+9.1.1) create infinite TTL shortened URLs
+9.1.2) create temporary TTL shortened URLs
+9.2) if ok - shorten the URL using our ALGO (below), record everything in the database (token, ttl, original url, shortened url, creation time, modification time, etc)
+10. return error response otherwise
+
+
+ALGO for shorten urls:
+considering the shortened url must be in format like https://[s.f0xx.org]/[3dfac0] where
+[s.f0xx.org] - the domain name of url shortener
+[3dfac0] - the shortened URL part
+it must be resolved finally in, for example, https://defender.net/very/long/link#with_anchors&param1=value1&paramN=valueN
+the database should keep the original url (https://defender.net/...) as well as the shortened (https://s.f0xx.org/3dfac0), so suggested database/table structure is (simplified):
+* id - auto-incremental int
+* token - token record that created this url, varchar(128)
+* signer - varchar(128) - trusted signer key
+* creation timestamp - datetime
+* modification timestamp - datetime
+* access timestamp - last access timestamp, datetime
+* origin - original url to shorten, text
+* shortened - text, self-explanatory
+* hash - text, self-explanatory (full hash, not a part of)
+*... TBD
+the hash for shorten suggested to be calculated as Java String hash (4-byte value in hex) taked the intermixed string as an input from URL and TOKEN, for example
+- URL is 'https://defender.net/very/long/link#with_anchors&param1=value1&paramN=valueN', TOKEN=4abef433d23ab5x&&83^c6
+- intermix, take 1 symbol from URL and 1 symbol from TOKEN, paste them:
+h t t p s : / / d e f e n d e r . n e t / v e r y / l o n g /
+ 4 a b e f 4 3 3 d 2 3 a b 5 x & & 8 3 ^ c 6 4 a b e f 4 3 3 d
+h4tatbpesf:4/3/3dde2f3eanbd5exr&.&n8e3t^/cv6e4rayb/elfo4n3g3/d - the string to calculate hash from
+                                            ^ here we have the interleaving of URL and TOKEN so starting the TOKEN from the beginning
+the similar logic with the URLs shorter than TOKENs:
+h t t p s : / / t . c o
+ 4 a b e f 4 3 3 d 2 3 a - taking just a part of TOKEN
+htta5bpesf:4/3/3td.2c3oa - the string to calculate hash from
+
+then we take 'the string to calculate hash from' and compute 4-8 bytes hex hash using Java string hashing algo. alternatively we can compute sha256 over it an cut only some part of sha256 result, like:
+htta5bpesf:4/3/3td.2c3oa | sha256 -> 0c7d8fd69bc54623b436fd9543489230c27f177a56383ea6a9aace9f87f04004
+take 12-20 bytes from '0c7d8fd69bc54623b436fd9543489230c27f177a56383ea6a9aace9f87f04004' -> 4623b436f
+so the shortened url would be https://s.f0xx.org/4623b436f
+
+The reverse lookup:
+is straightforward
+- BE takes a part of the request, 4623b436f for this example, and performs reverse lookup through the database, depending on hashing algo used. once found, returns the response with the full URL, error otherwise
+
+
+TBD and open questions list:
+
+Estimations:
+
+Severity: Medium