Better documentation, smoother checks

This commit is contained in:
Remy Moll 2022-06-25 16:06:45 +02:00
parent 12a7de91ed
commit 79e3f54955
3 changed files with 23 additions and 56 deletions
README.md
app/utils_check
env

@ -3,63 +3,28 @@
A utility to fetch article requests from slack and generate pdfs for them, fully automatically. A utility to fetch article requests from slack and generate pdfs for them, fully automatically.
## Running - Pure docker
> I recommend running with docker compose instead
### How to run - auto archiving mode
In this mode the program is launched as a docker container, in a headless mode. For persistence purposes a local storage volume is required, but that's it!
`docker run -it -v <your storage>:/app/file_storage/ auto_news`
You can specify additional parameters:
`docker run -it -v <your storage>:/app/file_storage/ auto_news debug` runs with debug values (does not write to prod db, does not send mails)
`docker run -it -v <your storage>:/app/file_storage/ auto_news upload` catches up on incomplete uploads to archive.
`docker run -it -v <your storage>:/app/file_storage/ auto_news reducedfetch` makes assumption about the status of the slack chat and greatly reduces the number of api calls (faster start up).
These parameters can be combined (mostyl for testing I guess)
Finally for manual file verification:
`docker run -it -v <your storage>:/app/file_storage/ -e DISPLAY=":0" --network host -v \$XAUTHORITY:/root/.Xauthority auto_news check` lets you visually verify the downloaded files. The additional parameters are required in order to open guis on the host.
### How to run - development mode
In this mode, a docker container is launched with an additional volume, the local code. You can test your code without the need to rebuild the image.
`docker run -it -v <your storage>:/app/file_storage/ -v <your code>:/code/ --entry-point /bin/bash auto_news`
You are droppped into a bash shell, in which you can navigate to the `/code` directory and then test live.
### Cheat-sheet Remy:
`docker run -it -v /mnt/Data/COSS/Downloads/auto_news.container/:/app/file_storage/ auto_news`
`docker run -it -v /mnt/Data/COSS/Downloads/auto_news.container/:/app/file_storage/ -v /mnt/Data/COSS/Development/auto_news/app:/code --entrypoint /bin/bash auto_news`
`docker run -it -v /mnt/Data/COSS/Downloads/auto_news.container/:/app/file_storage/ -e DISPLAY=":0" --network host -v XAUTHORITY:/root/.Xauthority auto_news check`
## Running - Docker compose ## Running - Docker compose
I also wrote a rudimentary docker compose file which makes running much more simple. Just run A rudimentary docker compose file makes for much simpler command linde calls.
`docker compose --env-file <desired mode> up` * For normal, `production` mode and `upload` mode, run:
`docker compose --env-file env/<desired mode> up`
All relevant passthroughs and mounts are specified through the env-file, for which I configured 4 versions: production, debug (development in general), upload and check. These files will have to be adapted to your individual setup but can be reused more easily. All relevant passthroughs and mounts are specified through the env-file, for which I configured 4 versions: production, debug (development in general), upload and check. These files will have to be adapted to your individual setup but can be reused more easily.
For the debug env-file, you will likely want interactivity, so you need to run: * For `debug` mode, you will likely want interactivity, so you need to run:
`docker compose --env-file env/debug up -d && docker compose --env-file env/debug exec auto_news bash && docker compose --env-file env/debug down` `docker compose --env-file env/debug up -d && docker compose --env-file env/debug exec auto_news bash && docker compose --env-file env/debug down`
which should automatically shutdown the containers once you are done. (`ctrl+d` to exit the container shell). If not, re-run `docker compose --env-file env/debug down` manually.
<!-- > Note: > Note:
> > The live-mounted code is then under `/code`. Note that the `DEBUG=true` environment variable is still set. If you want to test things on production, run `export DEBUG=false`. Running `python runner.py` will now run the newly written code but, with the production database and storage.
> The `debug` requires additional input. Once `docker compose up` is running, in a new session run `docker compose --env-file env/debug exec bash`. The live-mounted code is then under `/code`. Note that the `DEBUG=true` environment variable is still set. If you want to test things on production, run `export DEBUG=false`.
-->
* For `check` mode, some env-variables are also changed and you still require interactivity. You don't need the geckodriver service however. The simplest way is to run
`docker compose --env-file env/check run auto_news`
## Building ## Building
@ -71,9 +36,9 @@ In docker, simply run:
where the `Dockerfile` has to be in the working directory where the `Dockerfile` has to be in the working directory
In docker compose, run the usual command, but append In docker compose, run
`docker compose ... up --build` `docker compose --env-file env/production build`
@ -82,7 +47,6 @@ In docker compose, run the usual command, but append
## Roadmap: ## Roadmap:
[ ] automatically upload files to NAS :x: automatically upload files to NAS
[ ] handle paywalled sites like faz, spiegel, .. through their dedicated edu-friendly sites :x: handle paywalled sites like faz, spiegel, .. through their dedicated edu-friendly sites
...

@ -35,7 +35,7 @@ def file_overview(file_url: str, file_attributes: list, options: dict) -> None:
file_table = Table( file_table = Table(
title = file_url, title = file_url,
row_styles = ["white", "bright_black"], row_styles = ["white", "bright_black"],
min_width = 150 min_width = 100
) )
file_table.add_column("Attribute", justify = "right", no_wrap = True) file_table.add_column("Attribute", justify = "right", no_wrap = True)
@ -158,11 +158,11 @@ def verify_unchecked():
try: try:
# close any previously opened windows: # close any previously opened windows:
subprocess.call(["kill", "`pgrep evince`"]) # subprocess.call(["kill", "`pgrep evince`"])
os.system("pkill evince")
# then open a new one # then open a new one
subprocess.Popen(["evince", f"file://{os.path.join(article.save_path, article.file_name)}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) subprocess.Popen(["evince", f"file://{os.path.join(article.save_path, article.file_name)}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# supress evince gtk warnings # supress evince gtk warnings
print("done")
except Exception as e: except Exception as e:
print(e) print(e)
continue continue

3
env/check vendored

@ -11,3 +11,6 @@ CHECK=true
HEADLESS=true HEADLESS=true
UPLOAD=false UPLOAD=false
REDUCEDFETCH=false REDUCEDFETCH=false
# ENTRYPOINT="/bin/bash"
INTERACTIVE=true