Once Upon a Time in Spain
A bit more than a year ago I moved to Spain. Sunny beaches, tapas, siestas, and… the infamous Extranjería (immigration office).
To do anything official as a foreigner in Spain, you need an appointment, a cita previa. And getting one is like trying to catch a Legendary Pokémon with a regular Poké Ball. The website is always fully booked, slots disappear faster than free pizza in the office, and the process is about as user-friendly as Vim for first-time users. And no, I’m not exaggerating even a little bit, it’s a special kind of bureaucratic hell.
So there I am, refreshing the page like a maniac, hoping to get an appointment before my legal period to stay in the country turned into an impromptu hide-and-seek game with the authorities. Painful. Tedious. Hopeless… With the only dreaded message on the screen: “No hay citas disponibles” (No appointments available). Naturally, my inner developer whispered: “Dude, automate this!”
Dark Arts of Python in Conjunction with ChatGPT
The system literally requires you to camp on the website, hitting refresh until a slot magically appears. Why not write a piece of code to do this dirty work for me?
Now, I’m a backend Java developer. So my first thought was why not use familiar tools and spin up a Spring Boot application with a Selenium scenario? But let’s be real, that’s like bringing a bazooka to a knife fight. Python + Selenium, even though Python isn’t my sharpest sword, seemed like the most straightforward combo for a quick and dirty task.
And here comes my first experience vibe-coding with ChatGPT, which felt like pair-programming with a very caffeinated junior dev who never sleeps. The result? A working Selenium script that could:
- Open the Extranjería appointment page.
- Try to find available slots.
- Trigger further action if something pops up.
# Simplified Python Selenium snippet
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("https://sede.administracionespublicas.gob.es/icpplus/index.html")
# ... enters personal details ...
if "No hay citas" not in browser.page_source:
print("BINGO! There might be slots!")
Telegram, My Precious
Cool, the script works. But I didn’t want to keep an eye on terminal logs all day long. I needed alerts on the go. Enter the Telegram Bot API.
Creating a Telegram bot is as simple as chatting with @BotFather. He gives you a token, and with a simple HTTP request you can send messages to a Telegram chat.
I wired the script to shoot me a direct message whenever an appointment slot was found. That way, instead of constant anxiety, I just get a ding on Telegram meaning I need to take further action and check the immigration website.
# Simplified Python Selenium snippet
import requests
def send_telegram_notification(message):
bot_token = "BOT_TOKEN"
chat_id = "CHAT_ID"
url = f"https://api.telegram.org/bot{bot_token}/sendMessage"
payload = {"chat_id": chat_id, "text": message}
requests.post(url, json=payload)
# ... in the scraper ...
if appointment_found:
send_telegram_notification("Appointment found!")
I was living the dream. For a few days.
But What If I don’t have Access to my PC?
The script was running beautifully on my home PC. But guess what? I also like leaving my apartment. Traveling, riding my moto, you know, being outdoors.
So, here’s the snag I hit. During a short trip out of town, the script went silent. Later I discovered the Extranjería website had apparently blacklisted my PC making the script completely useless for a few days.
So… How do I run and tweak this script while on the move? Managing cloud stuff from the phone is clunky, and I wanted something lightweight and hackable.
That’s when the idea hit me: let’s run it on the phone itself!
Termux + proot-distro to the Rescue
My first thought was to try out whether Docker is a good fit for Android, but it felt like a rabbit hole I wasn’t ready to go down. Instead, I discovered the beautiful world of Termux and proot-distro.
Termux is a powerful terminal emulator for Android that gives you a Linux environment on your phone. Combined with proot-distro, it could be used to install a full-blown Ubuntu distribution. Under the hood, proot-distro uses user-space emulation to fake chroot without root access, which makes it possible to run entire Linux distros inside Termux.
With just a few commands, and I had Ubuntu running inside Termux:
pkg install proot-distro
proot-distro install ubuntu
proot-distro login ubuntu
Let’s face it - headless browsers get caught by modern websites, especially government ones, faster than you can say CAPTCHA. Using a full Linux desktop environment running via VNC server, I could run a real Firefox browser, making my script virtually undetectable:
apt install xfce4 tigervnc-standalone-server firefox
Then, the master shell script will orchestrate the whole thing: start a VNC server, launch the XFCE desktop, and then execute my Selenium script:
# Start VNC Server and XFCE
vncserver -localhost :1
# Run the scraper
DISPLAY=:1 python /path/to/my/script.py
The whole setup looks like this:
Simple, elegant, and effective, wouldn’t you agree?
The Empire (Android) Strikes Back
Everything was running smoothly… until Android decided to be Android. And a few hours later Termux was brutally killed by the system’s Phantom Process Killer. Even with battery optimizations turned off, and a WakeLock acquired for Termux, the OS decided my scrappy immigrant survival hack didn’t deserve to live.
If you’re on Android 12 or newer, you’ll almost certainly run into the phantom process killer. The solution is to disable it via an ADB shell command:
./adb shell "settings put global settings_enable_monitor_phantom_procs false"
But then came the next boss battle: the battery.
My initial design kept the VNC server and XFCE running continuously, with Selenium just chilling in a sleep loop between checks. VNC + XFCE desktop environment spawn tons of background processes - window manager, panel, notification daemons, and so on - which generates an extensive system resource burden. As a result constant memory pressure, along with the overhead of managing these processes, was enough to drain phone battery from 100% to “please send help” in a few hours.
The fix was elegantly simple. I moved the sleep logic out the Selenium script to the master shell script:
while true; do
# Start VNC Server and XFCE
vncserver -localhost :1
# Run the scraper
DISPLAY=:1 python /path/to/my/script.py
# Kill VNC server
vncserver -kill :1
# Sleep for a while before the next check
sleep 900 # 15 minutes
done
This way, the heavy desktop environment is only running when absolutely necessary, and the phone rests peacefully between checks.
Much better. Battery life saved. Sanity restored.
From Hack to a Framework
After this hack saved me and a few friends of mine from pulling our hair out over the Extranjería website, a thought sparked. That seems quite practical and broadly applicable, doesn’t it? So I polished the rough edges, bolted on some new features, and unleashed it upon the world as termux-web-scraper.
The best part? Minimal user interaction required. Just drop in your Selenium script - whether using the provided fluent API or not - and you’re ready to scrape!
# Define a scraping steps
def login(driver, state, notify):
send_keys(driver, ("id", "username"), "user")
send_keys(driver, ("id", "password"), "pass")
click_element(driver, ("id", "login-button"))
notify("Logged in successfully!")
# Build the scraper
scraper = (
ScraperBuilder()
.with_step("Login", login)
.with_notifier(TelegramNotifier("YOUR_BOT_TOKEN", "YOUR_CHAT_ID"))
.with_error_hook(ScreenshotErrorHook("./screenshots"))
.with_error_hook(NotificationErrorHook())
.build()
)
# Run the scraper
scraper.run()
I won’t put you to sleep with all the technical details here, but if you enjoy geeking out over the details, I highly recommend checking out the project documentation.
What about use cases? Appointment hunting, price monitoring, weather alerts… basically anything Selenium can handle.
SinoTrack Meets Telegram
Once you have a hammer, everything looks like a nail. What about my next nail?
My motorcycle’s SinoTrack GPS tracker’s alert system was stuck in the Stone Age, sending cryptic SMS messages that are as helpful as a screen door on a submarine.
So, I pointed my framework at it and built the sinotrack-alert-monitor. Now, instead of a lame SMS, I get a rich Telegram alert if my bike decides to go on an adventure without me, breaks a speed limit, or if the tracker just gives up the ghost.
And here’s the kicker: because it runs with Android + Termux, I have full control over the alerting system from my phone. I can turn it on and off or tweak it anytime. It’s like having a tiny, paranoid security guard for my bike, living right in my pocket.
Now my moto pings me on Telegram if something suspicious happens. More reliable, cheaper, and way geekier.
The Journey’s End (For Now)
The whole thing began because of poorly designed bureaucratic website, but it became a great learning opportunity and resulted in a useful system for automating tasks on a phone.
And speaking of automation, my co-pilot ChatGPT deserves some credit here too. Vibe-coding with it was a blast as much as helping me to brainstorm and refine parts of this very article!
The moral of the story: never underestimate the power of frustration + caffeine + geekery. Also, if you’re ever in Spain trying to book a cita previa with Extranjería, may the odds be ever in your favor.
Thank you for reading! I hope this journey has been as insightful for you as it was for me. If you have any questions, thoughts or would like to share your story, drop a comment or ping me - I’d love to hear it. Happy scraping!