Better documentation, smoother checks
This commit is contained in:
		
							
								
								
									
										64
									
								
								README.md
									
									
									
									
									
								
							
							
						
						
									
										64
									
								
								README.md
									
									
									
									
									
								
							@@ -3,63 +3,28 @@
 | 
				
			|||||||
A utility to fetch article requests from slack and generate pdfs for them, fully automatically.
 | 
					A utility to fetch article requests from slack and generate pdfs for them, fully automatically.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					
 | 
				
			||||||
## Running - Pure docker
 | 
					 | 
				
			||||||
> I recommend running with docker compose instead
 | 
					 | 
				
			||||||
### How to run - auto archiving mode
 | 
					 | 
				
			||||||
In this mode the program is launched as a docker container, in a headless mode. For persistence purposes a local storage volume is required, but that's it!
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v <your storage>:/app/file_storage/ auto_news`
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
You can specify additional parameters: 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v <your storage>:/app/file_storage/ auto_news debug` runs with debug values (does not write to prod db, does not send mails)
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v <your storage>:/app/file_storage/ auto_news upload` catches up on incomplete uploads to archive.
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v <your storage>:/app/file_storage/ auto_news reducedfetch` makes assumption about the status of the slack chat and greatly reduces the number of api calls (faster start up).
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
These parameters can be combined (mostyl for testing I guess)
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
Finally for manual file verification:
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v <your storage>:/app/file_storage/ -e DISPLAY=":0" --network host -v \$XAUTHORITY:/root/.Xauthority auto_news check` lets you visually verify the downloaded files. The additional parameters are required in order to open guis on the host.
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
### How to run - development mode
 | 
					 | 
				
			||||||
In this mode, a docker container is launched with an additional volume, the local code. You can test your code without the need to rebuild the image.
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v <your storage>:/app/file_storage/ -v <your code>:/code/ --entry-point /bin/bash auto_news`
 | 
					 | 
				
			||||||
You are droppped into a bash shell, in which you can navigate to the `/code` directory and then test live.
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
### Cheat-sheet Remy:
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v /mnt/Data/COSS/Downloads/auto_news.container/:/app/file_storage/ auto_news`
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v /mnt/Data/COSS/Downloads/auto_news.container/:/app/file_storage/ -v /mnt/Data/COSS/Development/auto_news/app:/code --entrypoint /bin/bash auto_news`
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
`docker run -it -v /mnt/Data/COSS/Downloads/auto_news.container/:/app/file_storage/ -e DISPLAY=":0" --network host -v XAUTHORITY:/root/.Xauthority auto_news check`
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
## Running - Docker compose 
 | 
					## Running - Docker compose 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
I also wrote a rudimentary docker compose file which makes running much more simple. Just run
 | 
					A rudimentary docker compose file makes for much simpler command linde calls. 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
`docker compose --env-file <desired mode> up`
 | 
					* For normal, `production` mode and `upload` mode, run:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					    `docker compose --env-file env/<desired mode> up`
 | 
				
			||||||
 | 
					
 | 
				
			||||||
    All relevant passthroughs and mounts are specified through the env-file, for which I configured 4 versions: production, debug (development in general), upload and check. These files will have to be adapted to your individual setup but can be reused more easily.
 | 
					    All relevant passthroughs and mounts are specified through the env-file, for which I configured 4 versions: production, debug (development in general), upload and check. These files will have to be adapted to your individual setup but can be reused more easily.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
For the debug env-file, you will likely want interactivity, so you need to run:
 | 
					* For `debug` mode, you will likely want interactivity, so you need to run:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
    `docker compose --env-file env/debug  up -d && docker compose --env-file env/debug exec auto_news bash && docker compose --env-file env/debug down`
 | 
					    `docker compose --env-file env/debug  up -d && docker compose --env-file env/debug exec auto_news bash && docker compose --env-file env/debug down`
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					    which should automatically shutdown the containers once you are done. (`ctrl+d` to exit the container shell). If not, re-run `docker compose --env-file env/debug down` manually.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
<!-- > Note:
 | 
					    > Note:
 | 
				
			||||||
>
 | 
					    > The live-mounted code is then under `/code`. Note that the `DEBUG=true` environment variable is still set. If you want to test things on production, run `export DEBUG=false`. Running `python runner.py` will now run the newly written code but, with the production database and storage.
 | 
				
			||||||
> The `debug` requires additional input. Once `docker compose up` is running, in a new session run `docker compose --env-file env/debug exec bash`. The live-mounted code is then under `/code`. Note that the `DEBUG=true` environment variable is still set. If you want to test things on production, run `export DEBUG=false`.
 | 
					 | 
				
			||||||
-->
 | 
					 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					* For `check` mode, some env-variables are also changed and you still require interactivity. You don't need the geckodriver service however. The simplest way is to run
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					`docker compose --env-file env/check run auto_news`
 | 
				
			||||||
 | 
					
 | 
				
			||||||
## Building
 | 
					## Building
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -71,9 +36,9 @@ In docker, simply run:
 | 
				
			|||||||
 | 
					
 | 
				
			||||||
where the `Dockerfile` has to be in the working directory
 | 
					where the `Dockerfile` has to be in the working directory
 | 
				
			||||||
 | 
					
 | 
				
			||||||
In docker compose, run the usual command, but append 
 | 
					In docker compose, run 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
`docker compose ... up --build`
 | 
					`docker compose --env-file env/production build`
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -82,7 +47,6 @@ In docker compose, run the usual command, but append
 | 
				
			|||||||
 | 
					
 | 
				
			||||||
## Roadmap:
 | 
					## Roadmap:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
[ ] automatically upload files to NAS
 | 
					:x: automatically upload files to NAS
 | 
				
			||||||
 | 
					
 | 
				
			||||||
[ ] handle paywalled sites like faz, spiegel, .. through their dedicated edu-friendly sites
 | 
					:x: handle paywalled sites like faz, spiegel, .. through their dedicated edu-friendly sites
 | 
				
			||||||
...
 | 
					 | 
				
			||||||
 
 | 
				
			|||||||
@@ -35,7 +35,7 @@ def file_overview(file_url: str, file_attributes: list, options: dict) -> None:
 | 
				
			|||||||
    file_table = Table(
 | 
					    file_table = Table(
 | 
				
			||||||
        title = file_url,
 | 
					        title = file_url,
 | 
				
			||||||
        row_styles = ["white", "bright_black"],
 | 
					        row_styles = ["white", "bright_black"],
 | 
				
			||||||
        min_width = 150
 | 
					        min_width = 100
 | 
				
			||||||
    )
 | 
					    )
 | 
				
			||||||
 | 
					
 | 
				
			||||||
    file_table.add_column("Attribute", justify = "right", no_wrap = True)
 | 
					    file_table.add_column("Attribute", justify = "right", no_wrap = True)
 | 
				
			||||||
@@ -158,11 +158,11 @@ def verify_unchecked():
 | 
				
			|||||||
        
 | 
					        
 | 
				
			||||||
        try:
 | 
					        try:
 | 
				
			||||||
            # close any previously opened windows:
 | 
					            # close any previously opened windows:
 | 
				
			||||||
            subprocess.call(["kill", "`pgrep evince`"])
 | 
					            # subprocess.call(["kill", "`pgrep evince`"])
 | 
				
			||||||
 | 
					            os.system("pkill evince")
 | 
				
			||||||
            # then open a new one
 | 
					            # then open a new one
 | 
				
			||||||
            subprocess.Popen(["evince", f"file://{os.path.join(article.save_path, article.file_name)}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 | 
					            subprocess.Popen(["evince", f"file://{os.path.join(article.save_path, article.file_name)}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 | 
				
			||||||
            # supress evince gtk warnings
 | 
					            # supress evince gtk warnings
 | 
				
			||||||
            print("done")
 | 
					 | 
				
			||||||
        except Exception as e:
 | 
					        except Exception as e:
 | 
				
			||||||
            print(e)
 | 
					            print(e)
 | 
				
			||||||
            continue
 | 
					            continue
 | 
				
			||||||
 
 | 
				
			|||||||
		Reference in New Issue
	
	Block a user