Compare commits

..

10 Commits

Author SHA1 Message Date
108f726d14 add license information 2024-06-15 21:47:18 -05:00
3848e1f777 it's a web interface 2024-05-20 05:21:13 -05:00
9680a416da move leglib into a python package 2024-05-19 17:56:26 -05:00
9ba154f654 add search to the setup 2024-05-19 17:51:51 -05:00
5c11ff4371 parser object overhauls
make conference names part of the parser object in preparation for the
implementation of billdb; refactor class structure for parsers
2024-05-19 16:02:33 -05:00
dbd9632e16 fix bill parsing in the main parser 2024-05-19 16:02:03 -05:00
11fbcb474a split up the parsers and other utilities 2024-05-03 13:49:16 -05:00
eabe1c98a0 add more changes to ledlib 2024-05-03 13:35:11 -05:00
a62453bdea to be squashed 2024-05-03 06:42:56 -05:00
875890a83a need to clean this up 2024-05-03 05:58:33 -05:00
77 changed files with 854 additions and 2124 deletions

View File

@ -1,3 +0,0 @@
DEBUG=1
SECRET_KEY=foo
DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]

View File

@ -1,2 +0,0 @@
SECRET_KEY=change_me
DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]

9
.gitignore vendored
View File

@ -1,9 +1,2 @@
*.log __pycache__/
*.pot
*.pyc *.pyc
__pycache__
db.sqlite3
media
uploads/
staticfiles/
.env.prod

View File

@ -1,6 +1,6 @@
# GNU AFFERO GENERAL PUBLIC LICENSE # GNU GENERAL PUBLIC LICENSE
Version 3, 19 November 2007 Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. Copyright (C) 2007 Free Software Foundation, Inc.
<https://fsf.org/> <https://fsf.org/>
@ -10,15 +10,17 @@ license document, but changing it is not allowed.
## Preamble ## Preamble
The GNU Affero General Public License is a free, copyleft license for The GNU General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure software and other kinds of works.
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast, to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to the GNU General Public License is intended to guarantee your freedom
share and change all versions of a program--to make sure it remains to share and change all versions of a program--to make sure it remains
free software for all its users. free software for all its users. We, the Free Software Foundation, use
the GNU General Public License for most of our software; it applies
also to any other work released this way by its authors. You can apply
it to your programs, too.
When we speak of free software, we are referring to freedom, not When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you price. Our General Public Licenses are designed to make sure that you
@ -27,34 +29,46 @@ them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things. free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights To protect your rights, we need to prevent others from denying you
with two steps: (1) assert copyright on the software, and (2) offer these rights or asking you to surrender the rights. Therefore, you
you this License which gives you legal permission to copy, distribute have certain responsibilities if you distribute copies of the
and/or modify the software. software, or if you modify it: responsibilities to respect the freedom
of others.
A secondary benefit of defending all users' freedom is that For example, if you distribute copies of such a program, whether
improvements made in alternate versions of the program, if they gratis or for a fee, you must pass on to the recipients the same
receive widespread use, become available for other developers to freedoms that you received. You must make sure that they, too, receive
incorporate. Many developers of free software are heartened and or can get the source code. And you must show them these terms so they
encouraged by the resulting cooperation. However, in the case of know their rights.
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to Developers that use the GNU GPL protect your rights with two steps:
ensure that, in such cases, the modified source code becomes available (1) assert copyright on the software, and (2) offer you this License
to the community. It requires the operator of a network server to giving you legal permission to copy, distribute and/or modify it.
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and For the developers' and authors' protection, the GPL clearly explains
published by Affero, was designed to accomplish similar goals. This is that there is no warranty for this free software. For both users' and
a different license, not a version of the Affero GPL, but Affero has authors' sake, the GPL requires that modified versions be marked as
released a new version of the Affero GPL which permits relicensing changed, so that their problems will not be attributed erroneously to
under this license. authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the
manufacturer can do so. This is fundamentally incompatible with the
aim of protecting users' freedom to change the software. The
systematic pattern of such abuse occurs in the area of products for
individuals to use, which is precisely where it is most unacceptable.
Therefore, we have designed this version of the GPL to prohibit the
practice for those products. If such problems arise substantially in
other domains, we stand ready to extend this provision to those
domains in future versions of the GPL, as needed to protect the
freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish
to avoid the special danger that patents applied to a free program
could make it effectively proprietary. To prevent this, the GPL
assures that patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and The precise terms and conditions for copying, distribution and
modification follow. modification follow.
@ -63,8 +77,7 @@ modification follow.
### 0. Definitions. ### 0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public "This License" refers to version 3 of the GNU General Public License.
License.
"Copyright" also means copyright-like laws that apply to other kinds "Copyright" also means copyright-like laws that apply to other kinds
of works, such as semiconductor masks. of works, such as semiconductor masks.
@ -533,47 +546,37 @@ from those to whom you convey the Program, the only way you could
satisfy both those terms and this License would be to refrain entirely satisfy both those terms and this License would be to refrain entirely
from conveying the Program. from conveying the Program.
### 13. Remote Network Interaction; Use with the GNU General Public License. ### 13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your
version supports such interaction) an opportunity to receive the
Corresponding Source of your version by providing access to the
Corresponding Source from a network server at no charge, through some
standard or customary means of facilitating copying of software. This
Corresponding Source shall include the Corresponding Source for any
work covered by version 3 of the GNU General Public License that is
incorporated pursuant to the following paragraph.
Notwithstanding any other provision of this License, you have Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work, License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version but the special requirements of the GNU Affero General Public License,
3 of the GNU General Public License. section 13, concerning interaction through a network will apply to the
combination as such.
### 14. Revised Versions of this License. ### 14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions The Free Software Foundation may publish revised and/or new versions
of the GNU Affero General Public License from time to time. Such new of the GNU General Public License from time to time. Such new versions
versions will be similar in spirit to the present version, but may will be similar in spirit to the present version, but may differ in
differ in detail to address new problems or concerns. detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program Each version is given a distinguishing version number. If the Program
specifies that a certain numbered version of the GNU Affero General specifies that a certain numbered version of the GNU General Public
Public License "or any later version" applies to it, you have the License "or any later version" applies to it, you have the option of
option of following the terms and conditions either of that numbered following the terms and conditions either of that numbered version or
version or of any later version published by the Free Software of any later version published by the Free Software Foundation. If the
Foundation. If the Program does not specify a version number of the Program does not specify a version number of the GNU General Public
GNU Affero General Public License, you may choose any version ever License, you may choose any version ever published by the Free
published by the Free Software Foundation. Software Foundation.
If the Program specifies that a proxy can decide which future versions If the Program specifies that a proxy can decide which future versions
of the GNU Affero General Public License can be used, that proxy's of the GNU General Public License can be used, that proxy's public
public statement of acceptance of a version permanently authorizes you statement of acceptance of a version permanently authorizes you to
to choose that version for the Program. choose that version for the Program.
Later license versions may give you additional or different Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any permissions. However, no additional obligations are imposed on any

View File

@ -1,19 +0,0 @@
prod: # execute this target on the production server in the nix-shell
cd franklincce; yes yes | python3 manage.py collectstatic
sh gen_kb.sh
sed "s|change_me|$(shell dd if=/dev/urandom bs=1024 count=1|base64)|g" .env.prod.orig > .env.prod
docker-compose -f docker-compose.prod.yml up -d --build
permissions: db.sqlite3
chmod -f 660 db.sqlite3
echo "make sure that db.sqlite3 is owned by group users"
db.sqlite3:
touch db.sqlite3
make_kb:
sh gen_kb.sh
clean:
rm -fr franklincce/staticfiles
docker-compose -f docker-compose.prod.yml down -v

View File

@ -1,49 +1,6 @@
yig yig
=== ---
`yig` is a django app for exploring bills and resolutions in previous YMCA CCE conferences. This is a parser for YMCA CCE YIG/MUN handbooks.
It's sort of ballooned to also contain a knowledge-base thing.
## Deploying in production See license information in `./LICENSE.md`.
0. Ensure that you have Docker installed and good to go.
For NixOS machines, this involves the following:
```nix
{ ... }
{
virtualisation.docker.enable = true;
users.users.<your user>.extraGroups = [ ... ] ++ [ "docker" ];
}
```
1. Enter the `nix-shell`.
2. `make permissions`
3. `make`
To tear down the docker container, type `make clean`.
If you've just started the instance, you also need to configure a superuser.
1. Run `docker ps | grep yig-web | awk -F' ' '{print $1}'` to get the container id
2. Run `docker exec -it <container-id> bash` to get a shell.
3. Run `python3 manage.py createsuperuser` and follow the prompts.
## Usage
In order to use the `explorer` component of this package, you need to add some legislative texts.
These are in the form of PDFs -- they come from the YMCA CCE website or are sometimes emailed or otherwise shared with you.
Only these PDFs will work because they follow a very specific format which the software exploits.
1. Login with your admin account.
2. Click on the `add` button next to `Legislation books` in the sidebar
3. Upload your book, add a name, and choose the correct `Import strategy`
4. Click save
## More information
More information about operation of this project can be found in the [knowledgebase](./kb) directory.
## License
`yig` is licensed under the AGPLv3 -- the terms of the license are available in [`LICENSE.md`](./LICENSE.md).

Binary file not shown.

99
cceexplorer/__init__.py Normal file
View File

@ -0,0 +1,99 @@
import secrets
from flask import Flask
from flask import render_template
from flask_bootstrap import Bootstrap
from .leglib.billdb import BillDB, BillQuery, QueryField, QueryAll
from .leglib.parsers import HSYIGPdfParser
parser = HSYIGPdfParser.from_filename(
filename="YIGVolunteerBook2024.pdf",
confname="HSVolunteer"
)
parser.parse()
db = BillDB()
db.add_conference(parser=parser)
def create_app(test_config=None):
app = Flask(__name__, instance_relative_config=True)
app.config.from_mapping(
SECRET_KEY=str(secrets.randbelow(100000000))
)
Bootstrap(app)
@app.route('/')
def index():
bills = db.search(query=QueryAll)
return render_template('index.html', number_bills=len(bills), number_conferences=2, bills=bills)
@app.route('/legislation/<conference>/<year>')
def show_conference(conference=QueryField.Any):
return conference
@app.route('/legislation/<conference>/<color>/<year>')
def show_color(
conference=QueryField.Any,
year=QueryField.Any,
color=QueryField.Any,
):
bills = db.search(query=BillQuery(
color=color,
year=int(year),
))
return render_template('color.html', bills=bills)
@app.route('/legislation/<conference>/<color>/<assembly>/<year>')
def show_assembly(
conference=QueryField.Any,
assembly=QueryField.Any,
color=QueryField.Any,
year=QueryField.Any,
):
bills = db.search(query=BillQuery(
color=color,
assembly=assembly,
year=int(year),
))
return render_template('assembly.html', bills=bills)
@app.route('/legislation/<conference>/<color>/<assembly>/<year>/<committee>')
def show_committee(
conference=QueryField.Any,
assembly=QueryField.Any,
color=QueryField.Any,
year=QueryField.Any,
committee=QueryField.Any,
):
bills = db.search(query=BillQuery(
color=QueryField.Any,
assembly=assembly,
year=int(year),
committee=int(committee),
))
return render_template('committee.html', bills=bills)
@app.route('/legislation/<conference>/<color>/<assembly>/<year>/<committee>/<order>')
def show_bill(
conference=QueryField.Any,
assembly=QueryField.Any,
color=QueryField.Any,
year=QueryField.Any,
committee=QueryField.Any,
order=QueryField.Any,
):
print(order, int(order))
print(color, assembly, year, committee, order)
bills = db.search(query=BillQuery(
color=color,
assembly=assembly,
year=int(year),
committee=int(committee),
order=int(order),
))
return render_template("bill.html", bill=bills[0])
return app

29
cceexplorer/analyser.py Normal file
View File

@ -0,0 +1,29 @@
from leglib.billdb import BillDB, BillQuery, QueryField, QueryAll
from leglib.parsers import HSYIGPdfParser
parser = HSYIGPdfParser.from_filename(
filename="YIGVolunteerBook2024.pdf",
confname="HSVolunteer"
)
parser.parse()
print(len(parser.bills))
db = BillDB()
db.add_conference(parser=parser)
allbills = len(db.search(query=QueryAll))
bluelen = len(db.search(query=BillQuery(color=QueryField.Colors.Blue)))
whitelen = len(db.search(query=BillQuery(color=QueryField.Colors.White)))
redlen = len(db.search(query=BillQuery(color=QueryField.Colors.Red)))
senatelen = len(db.search(query=BillQuery(assembly=QueryField.Assemblies.Senate)))
houselen = len(db.search(query=BillQuery(assembly=QueryField.Assemblies.House)))
franklincount = len(db.search(query=BillQuery(school="Franklin")))
print(allbills)
print(redlen, whitelen, bluelen, redlen + whitelen + bluelen)
print(senatelen, houselen, senatelen + houselen)
print(franklincount)

View File

@ -0,0 +1,128 @@
from .common import Bill, CCEColors, CCEAssemblies
from .parsers import BookParser
from typing import Type, Self
from dataclasses import dataclass
class QueryAny:
"""
Use this class to indicate an Any match for attributes without an Any attribute.
"""
pass
class SearchNotSatisified(BaseException):
pass
class QueryAll:
pass
class QueryField:
Any = object()
Colors = CCEColors
Assemblies = CCEAssemblies
@dataclass
class BillQuery:
"""
Holds a query for the BillDB.
"""
color: str | CCEColors | QueryField = QueryField.Any
assembly: str | CCEAssemblies | QueryField = QueryField.Any
committee: int | QueryField = QueryField.Any
year: int | QueryField = QueryField.Any
order: int | QueryField = QueryField.Any
subcommittee: str | QueryField = QueryField.Any
sponsors: str | QueryField = QueryField.Any
school: str | QueryField = QueryField.Any
bill_text: str | QueryField = QueryField.Any
title: str | QueryField = QueryField.Any
def __post_init__(self):
self.bill_text_concat = self.bill_text # for search compat reasons
class BillDB:
def __init__(self):
self.bills: list[Bill] = []
self.cache: dict[Bill]
@staticmethod
def code_enum_match(bill: Bill, query: BillQuery, attr: str) -> None:
"""
This is probably very slow. Maybe replace this with a better solution?
This function replaces repetitive code like this:
elif bill.assembly != CCEAssemblies.Any:
if bill.assembly != query.color:
raise SearchNotSatisified()
with this:
self.enum_match(bill, query, "color")
This is the case with exact_match and string_match, too.
"""
if query.__getattribute__(attr) == QueryField.Any:
return
# check the Any case
if query.__getattribute__(attr) != bill.code.__getattribute__(attr).__class__.Any:
# make sure we're not matching
if bill.code.__getattribute__(attr) != query.__getattribute__(attr):
raise SearchNotSatisified()
# if we do match, no exception
@staticmethod
def string_match(bill: Bill, query: BillQuery, attr: str) -> None:
"""
See self.code_enum_match for more info.
"""
if query.__getattribute__(attr) == QueryField.Any:
return
if not query.__getattribute__(attr).lower() in bill.__getattribute__(attr).lower():
raise SearchNotSatisified()
def add_conference(self: Self, parser: Type[BookParser]) -> None:
"""
Type[BookParser] -> any subclass of BookParser
"""
# this works because each BookParser must insert its self.confname into its self.bills[i].code.conference field.
self.bills += parser.bills
def search(self: Self, query: BillQuery | QueryAll) -> list[Bill]:
if query == QueryAll:
return self.bills
results = []
for bill in self.bills:
try:
# print("debug, q: {}, b: {}".format(str(query.committee), str(bill.code.committee)))
self.code_enum_match(bill, query, "color")
self.code_enum_match(bill, query, "assembly")
if not query.committee == QueryField.Any:
if not query.committee == bill.code.committee:
raise SearchNotSatisified()
if not query.order == QueryField.Any:
if not query.order == bill.code.docketplacement:
raise SearchNotSatisified()
if not query.committee == QueryField.Any:
if not query.year == bill.code.year:
raise SearchNotSatisified()
self.string_match(bill, query, "subcommittee")
self.string_match(bill, query, "sponsors")
self.string_match(bill, query, "school")
self.string_match(bill, query, "bill_text_concat")
self.string_match(bill, query, "title")
except SearchNotSatisified:
continue
results.append(bill)
return results

View File

@ -0,0 +1,161 @@
from enum import StrEnum, auto
class CCEColors(StrEnum):
Red = "Red"
White = "White",
Blue = "Blue",
Undefined = "Undefined", # some conferences don't have assemblies
Any = "Any" # for searching purposes
class CCEAssemblies(StrEnum):
Senate = "Senate",
House = "House",
GeneralAssembly = "GeneralAssembly",
Any = "Any" # for searching purposes
class BillCode:
def __init__(self, text: str):
# try to parse
# codes are in this rough format: "RSB/yy-c(c)-n(n)"
text = text.rstrip()
slashsplit = text.split('/')
dashsplit = slashsplit[1].split('-')
assemblycode = slashsplit[0]
self.color = assemblycode[0]
if self.color == "R":
self.color = CCEColors.Red
elif self.color == "W":
self.color = CCEColors.White
elif self.color == "B":
self.color = CCEColors.Blue
assemblydivision = assemblycode[1]
if assemblydivision == "S":
self.assembly = CCEAssemblies.Senate
elif assemblydivision == "H":
self.assembly = CCEAssemblies.House
elif assemblydivision == "G":
self.assembly = CCEAssemblies.GeneralAssembly
# reverse y2k problem; but conference years are stored in YY, not YYYY form
self.year = int(dashsplit[0]) + 2000
self.committee = int(dashsplit[1])
self.docketplacement = int(dashsplit[2])
self.stringrep = self.color[0].upper() + \
self.assembly[0].upper() + \
"B/{}-{}-{}".format(
str(self.year - 2000),
str(self.committee),
str(self.docketplacement)
)
self.conference: None | str = None # to be filled in with BookParser and friends
def __str__(self):
return "{} {} - {}-{}-{}".format(
self.color,
self.assembly,
str(self.year),
str(self.committee),
str(self.docketplacement)
)
class Bill:
def __init__(self,
code: str | BillCode,
sponsors: str,
subcommittee: str,
school: str,
bill_text: list[str],
title: str
):
if isinstance(code, str):
self.code = BillCode(code)
else:
self.code = code
self.sponsors = sponsors.rstrip()
self.subcommittee = subcommittee.rstrip()
self.school = school.rstrip()
self.bill_text = bill_text
self.title = title
@property
def bill_text_concat(self):
return ''.join(self.bill_text)
@property
def url(self):
if self.code.conference:
return "/legislation/" + '/'.join([
self.code.conference,
self.code.color,
self.code.assembly,
str(self.code.year),
str(self.code.committee),
str(self.code.docketplacement)
])
else:
return "/legislation/" + '/'.join([
"defaultconf",
self.code.color,
self.code.assembly,
str(self.code.year),
str(self.code.committee),
str(self.code.docketplacement)
])
@property
def committee_url(self):
if self.code.conference:
return "/legislation/" + '/'.join([
self.code.conference,
self.code.color,
self.code.assembly,
str(self.code.year),
str(self.code.committee)
])
else:
return "/legislation/" + '/'.join([
"defaultconf",
self.code.color,
self.code.assembly,
str(self.code.year),
str(self.code.committee)
])
@property
def assembly_url(self):
if self.code.conference:
return "/legislation/" + '/'.join([
self.code.conference,
self.code.color,
self.code.assembly,
str(self.code.year),
])
else:
return "/legislation/" + '/'.join([
"defaultconf",
self.code.color,
self.code.assembly,
str(self.code.year),
])
@property
def color_url(self):
if self.code.conference:
return "/legislation/" + '/'.join([
self.code.conference,
self.code.color,
str(self.code.year),
])
else:
return "/legislation/" + '/'.join([
"defaultconf",
self.code.color,
str(self.code.year),
])

20
cceexplorer/leglib/lib.py Normal file
View File

@ -0,0 +1,20 @@
class FitzBlockWrapper:
def __init__(self, block):
self.x0, self.y0, self.x1, \
self.y1, self.text, \
self.block_number, self.block_type = block
self.x0 = int(self.x0)
self.x1 = int(self.x1)
self.y0 = int(self.y0)
self.y1 = int(self.y1)
self.block_number = int(self.block_number)
self.block_type = int(self.block_type)
def __str__(self):
return str((
self.x0, self.y0, self.x1, self.y1, self.text
))
def __repl__(self):
return self.__str__()

View File

@ -0,0 +1,202 @@
import fitz
from typing import Any, Self, ClassVar
from itertools import groupby
from dataclasses import dataclass
from .lib import FitzBlockWrapper
from .common import Bill
@dataclass
class BookParser:
# class variables
humanname: ClassVar[str] = "Generic BookParser parent class."
description: ClassVar[str] = """
A generic description of the abilities of this BookParser.
"""
# everything else
document: fitz.Document
confname: str
@classmethod
def from_filename(cls, filename: str, confname: str):
return cls(
document=fitz.open(filename),
confname=confname
)
class HSYIGPdfParser(BookParser):
@staticmethod
def _words_in_superstring(words: list[str], superstring: str) -> bool:
for word in words:
if not str(word).lower() in str(superstring).lower():
return False
return True
def _generate_legislative_pages_list(self, sections: list[int]) -> list[int]:
"""
sections is an array of section pages plus the last page.
"""
current = 0
legislative_pages: list[int] = []
try:
while True:
legislative_pages += list(
range(
sections[current] + 1,
sections[current + 1],
1
)
)
current += 1
except IndexError:
pass
return legislative_pages
def _generate_section_markers(self, document: fitz.Document) -> list[int]:
section_pages = []
for page in document:
text = page.get_text().encode("utf8")
is_section_page = self._words_in_superstring(
words=[ "Committee", "YMCA", "Tennessee", "Youth", "in" ],
superstring=text
)
is_last_page = self._words_in_superstring(
words=[ "ABCs" ],
superstring=text
)
# print("page number {} contains sentintal? {}".format(page.number, is_section_page))
# if len(page.get_images()) == 3:
# print("page {} has one image!".format(page.number))
# print(page.get_images())
if is_section_page and len(page.get_images()) == 3:
section_pages.append(page.number)
if is_last_page and len(section_pages) > 2:
section_pages.append(page.number)
return section_pages
def _get_block_info_from_page(self, page: fitz.Page):
return [FitzBlockWrapper(i) for i in page.get_text("blocks")]
@staticmethod
def _remove_image_blocks(blocks: list[FitzBlockWrapper]) -> list[FitzBlockWrapper]:
to_return: list[FitzBlockWrapper] = []
for block in blocks:
if block.block_type == 0:
to_return.append(block)
return to_return
@staticmethod
def _remove_coordinate_information(blocks: list[FitzBlockWrapper]) -> list[FitzBlockWrapper]:
to_return: list[str] = []
for block in blocks:
to_return.append(block.text)
return to_return
@staticmethod
def _get_info_from_block(block, lat: int):
to_return = []
for i in block:
if math.floor(i[0]) == lat:
to_return.append(i)
return to_return
@staticmethod
def _split_list_by_element(arr: list[Any], pivot: Any):
output = []
current = []
for i in arr:
if i == pivot:
output.append(current)
current = []
else:
current.append(i)
output.append(current)
return output
def parse(self):
section_pages = self._generate_section_markers(self.document)
legislative_pages = self._generate_legislative_pages_list(section_pages)
joined_blocks: list[FitzBlockWrapper] = []
for page_number in legislative_pages:
page = self.document.load_page(page_number)
block_info = self._get_block_info_from_page(page)
joined_blocks += block_info[:-1] # remove the page number at the end of every page
joined_blocks = self._remove_image_blocks(joined_blocks)
joined_blocks = self._remove_coordinate_information(joined_blocks)
bill_header = joined_blocks[0]
splitted = self._split_list_by_element(joined_blocks, bill_header)
bills: list[Bill] = []
for splitted_item in splitted:
try:
bill_code, _, _, subcommittee, sponsors, school, *bill_text = splitted_item
except ValueError:
continue
bill_text = ' '.join(bill_text)
# print(type(bill_text))
pretty_printed = self._pretty_print_bill_text(bill_text)
bills.append(Bill(
code=bill_code,
subcommittee=subcommittee,
sponsors=sponsors,
school=school,
bill_text=pretty_printed["bill_array"],
title=pretty_printed["title"]
))
for bill in bills: # add the conference name to each
bill.code.conference = self.confname
self.bills = bills
@staticmethod
def _find_first_line_number(bill_arrays):
for i in range(len(bill_arrays)):
try:
if str(int(bill_arrays[i])) == bill_arrays[i]:
return i
except ValueError:
pass
def _pretty_print_bill_text(self, bill_text: str):
replaced = bill_text.replace("<EFBFBD>", "\n")
replaced = bill_text
replaced = replaced.split('\n')
replaced = [
i \
.replace('<EFBFBD>', ' ') \
.rstrip() \
.lstrip() \
for i in replaced
]
first_line_number = self._find_first_line_number(replaced)
title = ' '.join(replaced[:(first_line_number - 1)])
title = ' '.join(title.split()) # remove double spaces
rebuilt = replaced[first_line_number:][1::2]
# remove the last line number, it doesn't have a cooresponding space at the end
rebuilt = rebuilt[:-1]
# remove the first line, as it's the whitespace between the title and the bill text
rebuilt = rebuilt[1:]
return {
"title": title.lstrip(),
"bill_array": rebuilt
}

View File

@ -0,0 +1,14 @@
{% extends "base.html" %}
{% block title %} testing title {% endblock %}
{% block defcontent %}
<h1>{{ bills[0].code.color }} {{ bills[0].code.assembly }}</h1>
{% for bill in bills %}
<div class="container border-black">
<a href="{{ bill.url }}">({{bill.code.assembly[0]}}{{bill.code.committee}}/{{bill.code.docketplacement}}) {{ bill.title }}</a>
<p>Sponsors: {{ bill.sponsors }}</p>
<p>School: {{ bill.school }}</p>
</div>
{% endfor %}
{% endblock %}

View File

@ -0,0 +1,32 @@
{% extends "bootstrap/base.html" %}
{% block navbar %}
<nav class="navbar navbar-default">
<a class="navbar-brand" href="/">cceexplorer</a>
<div class="navbar-nav" id="navbarNav">
<ul class="nav navbar-nav">
<li class="nav-item">
<a class="nav-link" href="/">Home</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/search">Search</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/statistics">Statistics</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/conferences">Conferences</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/scores">Scores</a>
</li>
</ul>
</div>
</nav>
{% endblock %}
{% block content %}
<div class="container-fluid">
{% block defcontent %}{% endblock %}
</div>
{% endblock %}

View File

@ -0,0 +1,28 @@
{% extends "base.html" %}
{% block title %} cceexplorer - {{ bill.title }} {% endblock %}
{% block defcontent %}
<div class="container">
<div class="row">
<div class="col-xs-3 border border-dark rounded">
<h1>{{ bill.code.stringrep }}</h1>
<p><i>{{ bill.title }}</i></p>
<p>Introduced by {{ bill.sponsors }} (of {{ bill.school }}) within the {{ bill.subcommittee }} subcommittee</p>
<hr>
<ul>
<li><a href="{{ bill.committee_url }}">Go to this bill's committee</a></li>
<li><a href="{{ bill.assembly_url }}">Go to this bill's assembly</a></li>
<li><a href="{{ bill.color_url }}">Go to this bill's color grouping</a></li>
</ul>
</div>
<div class="col-xs-7 border border-dark rounded">
<br>
{% for line in bill.bill_text %}
{% if line == "" %}
{% endif %}
<p>{{ line }}</p>
{% endfor %}
</div>
</div>
</div>
{% endblock %}

View File

@ -0,0 +1,14 @@
{% extends "base.html" %}
{% block title %} testing title {% endblock %}
{% block defcontent %}
<h1>All {{bills[0].code.color}} Legislation</h1>
{% for bill in bills %}
<div class="container border-black">
<a href="{{ bill.url }}">({{bill.code.committee}}/{{bill.code.docketplacement}}) {{ bill.title }}</a>
<p>Sponsors: {{ bill.sponsors }}</p>
<p>School: {{ bill.school }}</p>
</div>
{% endfor %}
{% endblock %}

View File

@ -0,0 +1,14 @@
{% extends "base.html" %}
{% block title %} testing title {% endblock %}
{% block defcontent %}
<h1>{{ bills[0].code.assembly }} Committee {{ bills[0].code.committee }}</h1>
{% for bill in bills %}
<div class="container border-black">
<a href="{{ bill.url }}">({{bill.code.color}}) {{ bill.title }}</a>
<p>Sponsors: {{ bill.sponsors }}</p>
<p>School: {{ bill.school }}</p>
</div>
{% endfor %}
{% endblock %}

View File

@ -0,0 +1,15 @@
{% extends "base.html" %}
{% block title %} testing title {% endblock %}
{% block defcontent %}
<h1>{{ bills[0].code.color }} {{ bills[0].code.assembly }}</h1>
{% for bill in bills %}
<div class="container border-black">
<a href="{{ bill.url }}">({{bill.code.assembly[0]}}{{bill.code.committee}}/{{bill.code.docketplacement}}) {{ bill.title }}</a>
<p>Sponsors: {{ bill.sponsors }}</p>
<p>School: {{ bill.school }}</p>
</div>
{% endfor %}
{% endblock %}

View File

@ -0,0 +1,15 @@
{% extends "base.html" %}
{% block title %} testing title {% endblock %}
{% block defcontent %}
<h1>Welcome to cceexplorer</h1>
<p><i>an interactive database with {{ number_bills }} bills and {{ number_conferences }} conferences</i></p>
<p>here's all of them, down here!</p>
<ul>
{% for bill in bills %}
<li>
<a href="{{ bill.url }}">{{ bill.title }}</a>
</li>
{% endfor %}
</ul>
{% endblock %}

View File

@ -0,0 +1,6 @@
{% extends "base.html" %}
{% block title %} testing title {% endblock %}
{% block defcontent %}
<h1>testing</h1>
{% endblock %}

View File

@ -1,15 +0,0 @@
{ lib, config, pkgs, ... }:
{
# nix expressions to configure the relevant things
# manual nix-shell -c "make clean && make" (etc) is still needed, sadly
virtualisation.docker.enable = true;
services.nginx.virtualHosts."franklincce.beepboop.systems" = {
forceSSL = true;
enableACME = true;
locations."/" = {
proxyPass = "http://localhost:1337";
};
};
}

View File

@ -1,27 +0,0 @@
services:
web:
build:
context: ./franklincce
dockerfile: Dockerfile.prod
command: gunicorn franklincce.wsgi:application --bind 0.0.0.0:8000
volumes:
- type: volume
source: static_volume
target: /home/app/web/staticfiles
- type: bind
source: ./db.sqlite3
target: /home/app/web/db.sqlite3
expose:
- 8000
env_file:
- ./.env.prod
nginx:
build: ./nginx
volumes:
- static_volume:/home/app/web/staticfiles
ports:
- 1337:80
depends_on:
- web
volumes:
static_volume:

View File

@ -1,10 +0,0 @@
services:
web:
build: ./franklincce
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./franklincce/:/usr/src/app/
ports:
- 8000:8000
env_file:
- ./.env.dev

View File

@ -1,17 +0,0 @@
# pull official base image
FROM python:3.11.4-slim-buster
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# copy project
COPY . .

View File

@ -1,56 +0,0 @@
FROM python:3.11.4-slim-buster as builder
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install system dependencies
RUN apt-get update && \
apt-get install -y --no-install-recommends gcc
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt
# pull official base image
FROM python:3.11.4-slim-buster
# create directory for the app user
RUN mkdir -p /home/app
# create the app user
RUN addgroup --system app && adduser --system --group app
RUN usermod -a -G users app
# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/staticfiles
WORKDIR $APP_HOME
# install dependencies
RUN apt-get update && apt-get install -y --no-install-recommends netcat
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /usr/src/app/requirements.txt .
RUN pip install --upgrade pip
RUN pip install --no-cache /wheels/*
# copy entrypoint.prod.sh
COPY ./entrypoint.prod.sh .
RUN sed -i 's/\r$//g' $APP_HOME/entrypoint.prod.sh
RUN chmod +x $APP_HOME/entrypoint.prod.sh
# copy project
COPY . $APP_HOME
# chown all the files to the app user
RUN chown -R app:app $APP_HOME
# change to the app user
USER app
# run entrypoint.prod.sh
ENTRYPOINT ["/home/app/web/entrypoint.prod.sh"]

View File

@ -1,6 +0,0 @@
#!/bin/sh
python3 manage.py makemigrations
python3 manage.py migrate --run-syncdb
exec "$@"

View File

@ -1,21 +0,0 @@
from django.contrib import admin
from explorer import models
class LegislativeTextAdmin(admin.ModelAdmin):
list_display = ('__str__', 'legislation_title', 'school')
class LegislationBookAdmin(admin.ModelAdmin):
exclude = ("has_performed_export",)
to_register = [
[models.LegislativeText, LegislativeTextAdmin],
[models.LegislationBook, LegislationBookAdmin],
[models.LegislationClassification],
[models.School],
[models.Country],
[models.Sponsor],
[models.Category],
]
for i in to_register:
admin.site.register(*i)

View File

@ -1,6 +0,0 @@
from django.apps import AppConfig
class ExplorerConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'explorer'

View File

@ -1,327 +0,0 @@
from typing import Any, ClassVar
from dataclasses import dataclass
import fitz
class FitzBlockWrapper:
def __init__(self, block):
self.x0, self.y0, self.x1, \
self.y1, self.text, \
self.block_number, self.block_type = block
self.x0 = int(self.x0)
self.x1 = int(self.x1)
self.y0 = int(self.y0)
self.y1 = int(self.y1)
self.block_number = int(self.block_number)
self.block_type = int(self.block_type)
def __str__(self):
return str((
self.x0, self.y0, self.x1, self.y1, self.text
))
def __repl__(self):
return self.__str__()
def words_in_superstring(words: list[str], superstring: str) -> bool:
for word in words:
if not str(word).lower() in str(superstring).lower():
return False
return True
def split_by_lambda(arr: list[Any], func):
output = []
current = []
for item in arr:
if func(item):
output.append(current)
current = []
else:
current.append(item)
output.append(current)
return output
def get_block_by_x_value(arr: list[FitzBlockWrapper], xvalue: int) -> FitzBlockWrapper:
for item in arr:
if item.x0 == xvalue:
return item
def remove_block_by_x_value(arr: list[FitzBlockWrapper], xvalue: int) -> list[FitzBlockWrapper]:
return [i for i in arr if not i.x0 == xvalue]
class CCEParserBase():
section_page_words: ClassVar[list[str]]
last_page_words: ClassVar[list[str]]
split_leg_pages_needle: ClassVar[str]
def __init__(self, document: fitz.Document):
self.document = document
self.__post_init__()
def __post_init__(self):
# run all the processing steps here
self.parse_legislative_metablocks()
def generate_section_markers(self) -> list[int]:
"""
In the YIG/MUN manuals, there's section markers that delineate between the different
committees within the manual. Let's find those, and then the last legislative page.
"""
section_pages = []
for page in self.document:
text = page.get_text().encode("utf8")
is_section_page = words_in_superstring(
words = self.section_page_words,
superstring = text
)
is_last_page = words_in_superstring(
words = self.last_page_words,
superstring = text
)
if is_section_page:
section_pages.append(page.number)
if is_last_page and len(section_pages) > 2:
section_pages.append(page.number)
return section_pages
def get_legislative_pages(self):
"""
Generate the section markers, then fill in the pages between them.
"""
current = 0
sections = self.generate_section_markers()
legislative_pages: list[int] = []
try:
while True:
legislative_pages += list(
range(
sections[current] + 1,
sections[current + 1],
1
)
)
current += 1
except IndexError:
pass
return legislative_pages
def concat_blocks_for_leg_pages(self):
"""
From the legislative pages, concatenate the "blocks" of text in the PDF.
"""
blocks = []
pages = [self.document.load_page(page_num) for page_num in self.get_legislative_pages()]
for page in pages:
block_info = [FitzBlockWrapper(block) for block in page.get_text("blocks")]
blocks += block_info
return blocks
def split_leg_pages(self):
"""
We have the collection of legislative page text blocks. We need
to split them now. We split on the text "71st General Assembly...
Youth in Government"
"""
blocks = self.concat_blocks_for_leg_pages()
# each item within splitted is called a "legislative meta-block"
splitted = split_by_lambda(blocks, lambda x: self.split_leg_pages_needle in x.text)
return splitted[1:] # there's an empty array at the beginning
def handle_the_rest(self, the_rest):
weird_character = u''
another_weird_character = u'\uFFFd'
splitted_by_weird = the_rest.replace(weird_character, another_weird_character).split(another_weird_character)
title_content = ''.join(
splitted_by_weird[0].split('\n')[:-1]
).rstrip().lstrip()
bill_text = [i.split('\n')[0][1:] for i in splitted_by_weird[1:]]
return {
"bill_text": '\n'.join(bill_text),
"title": title_content
}
def parse_legislative_metablocks(self):
output = []
splitted = self.split_leg_pages()
for legislative_text in splitted:
# there are some blocks that contain just one value
# and are aligned to some x value on the pdf
# it's an easy way to extract stuff
try:
country = get_block_by_x_value(legislative_text, 139).text.rstrip()
country = country.replace("Sponsor: ", "").lstrip()
except AttributeError:
country = None # this is a yig bill
try:
category = get_block_by_x_value(legislative_text, 151).text.rstrip().lstrip()
except AttributeError:
try:
category = get_block_by_x_value(legislative_text, 153).text.rstrip().lstrip()
except AttributeError:
print([(i.text, i.x0) for i in legislative_text])
leg_code = get_block_by_x_value(legislative_text, 88).text.rstrip()
try:
school = get_block_by_x_value(legislative_text, 177).text.rstrip().lstrip()
except AttributeError:
try:
school = get_block_by_x_value(legislative_text, 186).text.rstrip().lstrip()
except AttributeError:
school = "you tell me, man"
try:
sponsors = get_block_by_x_value(legislative_text, 163).text.rstrip().lstrip()
except AttributeError:
try:
sponsors = get_block_by_x_value(legislative_text, 166).text.rstrip().lstrip()
except AttributeError:
sponsors = "you tell me, man"
the_rest = ''.join([i.text for i in legislative_text[12:]])
handled = self.handle_the_rest(the_rest)
title = handled["title"]
bill_text = handled["bill_text"]
codesplit = leg_code.split('/')
assembly = codesplit[0]
dashsplit = codesplit[1].split('-')
year = 2000 + int(dashsplit[0])
committee = int(dashsplit[1])
docket_order = int(dashsplit[2])
output.append({
"assembly": assembly,
"year": year,
"committee": committee,
"docket_order": docket_order,
"category": category,
"country": country,
"school": school,
"sponsors": sponsors,
"legislation_title": title,
"text": bill_text
})
self.output = output
class HSMUN23(CCEParserBase):
section_page_words = ["Committee", "Model", "United", "YMCA", "Tennessee", "Nations"]
last_page_words = ["ABCs"]
split_leg_pages_needle = "43rd General Assembly"
class HSYIG24(CCEParserBase):
section_page_words = [ "Committee", "YMCA", "Tennessee", "Youth", "in" ]
last_page_words = [ "ABCs" ]
split_leg_pages_needle = "71st General Assembly"
def generate_section_markers(self) -> list[int]:
"""
This overrides the regular method because we need to check
for three images on a section page
"""
section_pages = []
for page in self.document:
text = page.get_text().encode("utf8")
is_section_page = words_in_superstring(
words = self.section_page_words,
superstring = text
)
is_last_page = words_in_superstring(
words = self.last_page_words,
superstring = text
)
if is_section_page and len(page.get_images()) == 3:
section_pages.append(page.number)
if is_last_page and len(section_pages) > 2:
section_pages.append(page.number)
return section_pages
def parse_legislative_metablocks(self):
"""
This is YIG specific code
"""
output = []
splitted = self.split_leg_pages()
for legislative_text in splitted:
# there are some blocks that contain just one value
# and are aligned to some x value on the pdf
# it's an easy way to extract stuff
legislative_text = remove_block_by_x_value(legislative_text, 565) # remove page numbers
category = get_block_by_x_value(legislative_text, 139).text.rstrip().lstrip()
leg_code = get_block_by_x_value(legislative_text, 88).text.rstrip()
school = get_block_by_x_value(legislative_text, 163).text.rstrip().lstrip()
sponsors = get_block_by_x_value(legislative_text, 152).text.rstrip().lstrip()
the_rest = ''.join([i.text for i in legislative_text[6:]])
handled = self.handle_the_rest(the_rest)
title = handled["title"]
bill_text = handled["bill_text"]
codesplit = leg_code.split('/')
assembly = codesplit[0]
dashsplit = codesplit[1].split('-')
year = 2000 + int(dashsplit[0])
committee = int(dashsplit[1])
docket_order = int(dashsplit[2])
output.append({
"assembly": assembly,
"year": year,
"committee": committee,
"docket_order": docket_order,
"category": category,
"country": None, # this is a yig bill
"school": school,
"sponsors": sponsors,
"legislation_title": title,
"text": bill_text
})
self.output = output
def main():
argv = sys.argv
doc = fitz.open(argv[1])
if argv[2] == "HSYIG":
doc = HSYIG24(doc)
elif argv[2] == "HSMUN":
doc = HSMUN23(doc)
else:
print("nonvalid book thing")
return
for text in doc.output:
print("{} {} {} ---------------------------- {}".format(
text["country"], text["category"],
text["legislation_title"], text["text"]
))
if __name__ == "__main__":
import sys
main()

View File

@ -1,87 +0,0 @@
# Generated by Django 4.2.14 on 2024-07-24 06:38
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='Country',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=256)),
],
options={
'verbose_name_plural': 'Countries',
},
),
migrations.CreateModel(
name='LegislationBook',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('conference_type', models.CharField(choices=[('M', 'Middle School'), ('H', 'High School')], default='H', max_length=1)),
('pdf', models.FileField(upload_to='uploads/')),
('name', models.CharField(max_length=256)),
('import_strategy', models.CharField(choices=[('HSYIGBookParser', 'High School YIG Book Parser 1'), ('HSMUNBookParser', 'High School MUN Book Parser 1')], default='HSYIGBookParser', max_length=128)),
('has_performed_export', models.BooleanField(default=False)),
],
options={
'verbose_name': 'Book',
'verbose_name_plural': 'Books',
},
),
migrations.CreateModel(
name='LegislationClassification',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(help_text='Name of this classification.', max_length=256)),
('text_to_match', models.CharField(help_text='a comma seperated list of keywords to include in the classification. spaces and dashes are discluded.', max_length=256)),
],
options={
'verbose_name': 'Topic',
'verbose_name_plural': 'Topics',
},
),
migrations.CreateModel(
name='School',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=256)),
],
),
migrations.CreateModel(
name='Sponsor',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=256)),
],
),
migrations.CreateModel(
name='LegislativeText',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('assembly', models.CharField(choices=[('RGA', 'Red General Assembly'), ('BGA', 'Blue General Assembly'), ('WGA', 'White General Assembly'), ('RHB', 'Red House'), ('BHB', 'Blue House'), ('WHB', 'White House'), ('RSB', 'Red Senate'), ('BSB', 'Blue Senate'), ('WSB', 'White Senate'), ('SEN', 'Senate'), ('HOU', 'House'), ('GEN', 'General Assembly')], default='GEN', max_length=3)),
('text', models.TextField()),
('year', models.IntegerField()),
('committee', models.IntegerField()),
('category', models.CharField(max_length=256)),
('docket_order', models.IntegerField()),
('legislation_title', models.CharField(max_length=512)),
('country', models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, to='explorer.country')),
('from_book', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='explorer.legislationbook')),
('school', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='explorer.school')),
('sponsors', models.ManyToManyField(blank=True, to='explorer.sponsor')),
],
options={
'verbose_name': 'Legislation',
'verbose_name_plural': 'Legislation',
},
),
]

View File

@ -1,26 +0,0 @@
# Generated by Django 4.2.14 on 2024-07-24 17:58
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('explorer', '0001_initial'),
]
operations = [
migrations.CreateModel(
name='Category',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=256)),
],
),
migrations.AlterField(
model_name='legislativetext',
name='category',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='explorer.category'),
),
]

View File

@ -1,234 +0,0 @@
from django.db import models
from django.utils.translation import gettext_lazy as _
from django.urls import reverse
from .leglib import HSYIG24, HSMUN23
import io
import fitz
from collections import namedtuple
def InstantiateIfNone(model, name):
"""
Search the model for instances by name.
If there's none, then create one.
"""
filtered = model.objects.filter(name__exact=name)
try:
return filtered[0]
except IndexError:
obj = model(name=name)
obj.save()
return obj
class School(models.Model):
name = models.CharField(max_length=256)
def __str__(self):
return self.name
def get_absolute_url(self):
our_name = __class__.__name__
return reverse("{}.detail".format(our_name), kwargs={"model_id": self.id})
class Country(models.Model):
name = models.CharField(max_length=256)
class Meta:
verbose_name_plural = "Countries"
def __str__(self):
return self.name
def get_absolute_url(self):
our_name = __class__.__name__
return reverse("{}.detail".format(our_name), kwargs={"model_id": self.id})
class Sponsor(models.Model):
name = models.CharField(max_length=256)
def __str__(self):
return self.name
def get_absolute_url(self):
our_name = __class__.__name__
return reverse("{}.detail".format(our_name), kwargs={"model_id": self.id})
class Category(models.Model):
name = models.CharField(max_length=256)
class Meta:
verbose_name_plural = "Categories"
def __str__(self):
return "{}".format(self.name)
def get_absolute_url(self):
our_name = __class__.__name__
return reverse("{}.detail".format(our_name), kwargs={"model_id": self.id})
class LegislationBook(models.Model):
class Meta:
verbose_name = "Book"
verbose_name_plural = "Books"
class ConferenceType(models.TextChoices):
MIDDLE = "M", _("Middle School")
HIGH = "H", _("High School")
class ImportStrategy(models.TextChoices):
HSYIGA = "HSYIGBookParser", _("High School YIG Book Parser 1")
HSMUNA = "HSMUNBookParser", _("High School MUN Book Parser 1")
conference_type = models.CharField(
max_length=1,
choices=ConferenceType.choices,
default=ConferenceType.HIGH,
)
pdf = models.FileField(upload_to="uploads/")
name = models.CharField(max_length=256)
import_strategy = models.CharField(
max_length=128,
choices=ImportStrategy.choices,
default=ImportStrategy.HSYIGA
)
has_performed_export = models.BooleanField(default=False)
def save(self, **kwargs):
if not self.has_performed_export:
self.has_performed_export = True
super().save(**kwargs)
else:
return
the_file = io.BytesIO(self.pdf.file.file.read())
the_document = fitz.open(stream=the_file)
if self.import_strategy == "HSYIGBookParser":
parsed = HSYIG24(the_document)
elif self.import_strategy == "HSMUNBookParser":
parsed = HSMUN23(the_document)
else:
return
for text in parsed.output:
text["school"] = InstantiateIfNone(School, text["school"])
if text["country"]:
# there's sometimes "Dominican Republic" and "Dominican Republic 2"
# handle that gracefully
text["country"] = text["country"].replace(" 2", "")
text["country"] = InstantiateIfNone(Country, text["country"])
if not text["category"] or text["category"] == "Select One--":
text["category"] = "No category"
text["category"] = InstantiateIfNone(Category, text["category"])
sponsors = text["sponsors"].split(', ')
sponsors = [InstantiateIfNone(Sponsor, sponsor) for sponsor in sponsors]
del text["sponsors"]
text = LegislativeText(**text, from_book=self)
text.save()
for sponsor in sponsors:
text.sponsors.add(sponsor)
def __str__(self):
return "{}".format(self.name)
def get_absolute_url(self):
our_name = __class__.__name__
return reverse("{}.detail".format(our_name), kwargs={"model_id": self.id})
class LegislativeText(models.Model):
class Meta:
verbose_name = "Legislation"
verbose_name_plural = "Legislation"
class Assemblies(models.TextChoices):
RGA = "RGA", _("Red General Assembly")
BGA = "BGA", _("Blue General Assembly")
WGA = "WGA", _("White General Assembly")
RHB = "RHB", _("Red House")
BHB = "BHB", _("Blue House")
WHB = "WHB", _("White House")
RSB = "RSB", _("Red Senate")
BSB = "BSB", _("Blue Senate")
WSB = "WSB", _("White Senate")
SEN = "SEN", _("Senate")
HOU = "HOU", _("House")
GEN = "GEN", _("General Assembly")
assembly = models.CharField(
max_length=3,
choices=Assemblies.choices,
default=Assemblies.GEN
)
text = models.TextField()
year = models.IntegerField()
committee = models.IntegerField()
category = models.ForeignKey(Category, on_delete=models.CASCADE)
docket_order = models.IntegerField()
school = models.ForeignKey(School, on_delete=models.CASCADE)
sponsors = models.ManyToManyField(Sponsor, blank=True)
from_book = models.ForeignKey(LegislationBook, on_delete=models.CASCADE)
legislation_title = models.CharField(max_length=512)
country = models.ForeignKey(Country, on_delete=models.CASCADE, null=True)
def __str__(self):
return "{}/{}-{}-{}".format(
self.assembly,
str(self.year),
self.committee,
self.docket_order,
)
def get_lines(self):
cls = namedtuple('LegLine', ['linenumber', 'linetext'])
return [cls(i + 1, j) for i, j in enumerate(self.text.split('\n'))]
def is_bill(self):
if self.assembly in [
"RHB",
"BHB",
"WHB",
"RSB",
"BSB",
"WSB",
"SEN",
"HOU",
]:
return True
return False
def is_resolution(self):
if self.assembly in ["RGA", "BGA", "WGA", "GEN"]:
return True
return False
class LegislationClassification(models.Model):
class Meta:
verbose_name = "Topic"
verbose_name_plural = "Topics"
name = models.CharField(max_length=256, help_text="Name of this classification.")
text_to_match = models.CharField(
max_length=256,
help_text="a comma seperated list of keywords to include in the classification. spaces and dashes are discluded."
)
def __str__(self):
return "{}".format(self.name)
def get_absolute_url(self):
our_name = __class__.__name__
return reverse("{}.detail".format(our_name), kwargs={"model_id": self.id})
models_in_index = [
LegislationClassification,
School,
Country,
Sponsor,
Category,
LegislationBook
]

View File

@ -1,17 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>All legislative texts</h1>
{% if legislative_texts %}
<ul>
{% for text in legislative_texts %}
<li><a href="{% url 'viewleg' text.id %}">{{ text.legislation_title }}</a></li>
{% endfor %}
</ul>
{% else %}
<p>No texts available</p>
{% endif %}
</div>
{% endblock content %}

View File

@ -1,30 +0,0 @@
<!DOCTYPE html>
<html>
<head>
{% if page_title %}
<title>explorer - {{ title }}</title>
{% else %}
<title>explorer</title>
{% endif %}
<link rel="stylesheet" href="/static/style.css">
</head>
<body>
<nav id="navbar" class="boxed">
<div id="leftnav">
<a href="/explorer">explorer</a>
<a href="/knowledge">knowledge</a>
</div>
<div id="rightnav">
<a href="/explorer/all">all</a>
<a href="/explorer/groups">by group</a>
<a href="/explorer/search">search</a>
<a href="/explorer/stats">stats</a>
</div>
</nav>
{% block content %}
{% endblock content %}
</body>
</html>

View File

@ -1,15 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>View legislation</h1>
<ul>
{% for name, url in listing.items %}
<li><a href="{{ url }}">By {{ name }}</a></li>
{% endfor %}
</ul>
</div>
{% endblock content %}

View File

@ -1,15 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>All topics</h1>
<ul>
{% for topic in classifications %}
<li><a href="/explorer/topics/{{ topic.id }}">{{ topic.name }}</a></li>
{% endfor %}
</ul>
</div>
{% endblock content %}

View File

@ -1,38 +0,0 @@
{% comment %}
This is a component -- I'm only using this because:
1. I'm too lazy to install django-components, and
2. This is the only time a component is necessary, so it doesn't make
any sense to take on an extra dependancy.
{% endcomment %}
<div class="legcomponent">
<a href="/explorer/legislation/{{ legislation.id }}"><h2 class="legtitle">{{ legislation.legislation_title }}</h2></a>
<div class="legmetadata">
<p>
<i>{{ legislation.assembly }}/{{ legislation.committee }}/{{ legislation.docket_order }}</i>
&middot;
{% for sponsor in legislation.sponsors.all %}
<a href="/explorer/sponsors/{{ sponsor.id }}">{{ sponsor.name }}</a>{% if not forloop.last %}, {% endif %}
{% endfor %}
&middot;
<a href="/explorer/schools/{{ legislation.school.id }}">{{ legislation.school }}</a>
&middot;
{% if legislation.country %}
<a href="/explorer/countries/{{ legislation.country.id }}">{{ legislation.country }}</a>
&middot;
{% endif %}
<a href="/explorer/categories/{{ legislation.category.id }}">{{ legislation.category }}</a>
&middot;
<a href="/explorer/conference/{{ legislation.from_book.id }}">{{ legislation.from_book.name }}</a>
</p>
</div>
</div>

View File

@ -1,19 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
{% if sample.is_bill %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
{% elif sample.is_resolution %}
<link rel="stylesheet" type="text/css" href="/static/un.css" />
{% endif %}
<div class="boxed">
<h1>{{ book.name }} Conference</h1>
<ul>
{% for text in legislation %}
<li><a href="/explorer/legislation/{{ text.id }}">{{ text.legislation_title }}</a></li>
{% endfor %}
</ul>
</div>
{% endblock content %}

View File

@ -1,32 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>Welcome to explorer</h1>
<p><i>an interactive database for YMCA CCE legislation</i></p>
<h2>Some randomly selected legislation</h2>
{% if legislation %}
{% autoescape off %}
<ul>
{% for text in legislation %}
{{ text }}
{% endfor %}
</ul>
{% endautoescape %}
{% else %}
<p>No texts available. If you're the admin, you need to add some texts with the admin interface.</p>
{% endif %}
<h2>About this instance</h2>
<p>
This is a self hosted instance of <i>explorer</i>, a Django web app for exploring resolutions and bills in YMCA CCE conferences.
It's licensed under the AGPL.
The source code is available <a href="https://git.beepboop.systems/stupidcomputer/yig">here</a> and <a href="https://github.com/stupidcomputer/yig">here</a>.
</p>
</div>
{% endblock content %}

View File

@ -1,43 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
{% if legislation.is_bill %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
{% elif legislation.is_resolution %}
<link rel="stylesheet" type="text/css" href="/static/un.css" />
{% endif %}
<div id="legcontainer">
<div id="leginfo" class="boxed">
<h1 class="no-margin-top">{{ legislation.legislation_title }}</h1>
<p><i>{{ legislation.assembly }}/{{ legislation.committee }}/{{ legislation.docket_order }}</i></p>
<p>Sponsored by
{% for sponsor in legislation.sponsors.all %}
<a href="/explorer/sponsors/{{ sponsor.id }}">{{ sponsor.name }}</a>{% if not forloop.last %}, {% endif %}
{% endfor %}
of <a href="/explorer/schools/{{ legislation.school.id }}">{{ legislation.school }}</a></p>
{% if legislation.country %}
<p>The delegates above represented the <a href="/explorer/countries/{{ legislation.country.id }}">Delegation of {{ legislation.country }}</a>.</p>
{% endif %}
<p>This legislation was filed in the <a href="/explorer/categories/{{ legislation.category.id }}">{{ legislation.category }}</a> category</p>
<p>Presented as part of the <a href="/explorer/conference/{{ legislation.from_book.id }}">{{ legislation.from_book.name }}</a> conference</p>
</div>
<div id="legislation" class="boxed">
<table>
{% for line in lines %}
<tr>
<td class="legnumbers">{{ line.linenumber }}</td>
<td>{{ line.linetext }}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock content %}

View File

@ -1,15 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>{{ result_name }}</h1>
<ul>
{% for instance in instances %}
<li><a href="{{ instance.get_absolute_url }}">{{ instance.name }}</a></li>
{% endfor %}
</ul>
</div>
{% endblock content %}

View File

@ -1,17 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>{{ result_name }}</h1>
{% autoescape off %}
<ul>
{% for text in legislation %}
{{ text }}
{% endfor %}
</ul>
{% endautoescape %}
</div>
{% endblock content %}

View File

@ -1,15 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>Search legislation by keyword</h1>
<form action="/explorer/search" method="get">
<input type="text" id="search_term" name="search_term" />
<button type="submit">Execute search</button>
<p>
You can use this page to search through legislation by keyword. The search is case sensitive, so be careful when pressing your shift key.
</p>
</form>
</div>
{% endblock %}

View File

@ -1,28 +0,0 @@
{% extends "explorer/base.html" %}
{% block content %}
<link rel="stylesheet" type="text/css" href="/static/tn.css" />
<div class="boxed">
<h1>Explorer statistics</h1>
<p>Total pieces of legislation: {{ all }}</p>
<p>Red Senate Bills: {{ red_senate }}</p>
<p>White Senate Bills: {{ white_senate }}</p>
<p>Blue Senate Bills: {{ blue_senate }}</p>
<p>Red House Bills: {{ red_house }}</p>
<p>White House Bills: {{ white_house }}</p>
<p>Blue House Bills: {{ blue_house }}</p>
<p>Red General Assembly Resolutions: {{ red_ga }}</p>
<p>White General Assembly Resolutions: {{ white_ga }}</p>
<p>Blue General Assembly Resolutions: {{ blue_ga }}</p>
</div>
{% endblock content %}

View File

@ -1,3 +0,0 @@
from django.test import TestCase
# Create your tests here.

View File

@ -1,26 +0,0 @@
from django.urls import path
from . import views
urlpatterns = [
path("", views.index, name="index"),
path("all/", views.all, name="all"),
path("stats/", views.stats, name="stats"),
path("legislation/<int:legislation_id>/", views.view_legislation, name="viewleg"),
path("topics/<int:model_id>/", views.get_all_classified_by_id, name="LegislationClassification.detail"),
path("topics/", views.get_all_classifications, name="LegislationClassification"),
path("search/", views.handle_search, name="search_legislation"),
# these are named weirdly -- see models.py School and Country definitions
path("schools/<int:model_id>/", views.get_all_by_school, name="School.detail"),
path("countries/<int:model_id>/", views.get_all_by_country, name="Country.detail"),
path("sponsors/<int:model_id>/", views.get_all_by_sponsor, name="Sponsor.detail"),
path("categories/<int:model_id>/", views.get_all_by_category, name="Category.detail"),
path("conference/<int:model_id>/", views.get_all_by_conference, name="LegislationBook.detail"),
path("schools/", views.get_all_schools, name="School"),
path("countries/", views.get_all_countries, name="Country"),
path("groups/", views.return_groups, name="Groups"),
path("sponsors/", views.get_all_sponsors, name="Sponsor"),
path("categories/", views.get_all_categories, name="Category"),
path("conference/", views.get_all_conferences, name="LegislationBook"),
]

View File

@ -1,184 +0,0 @@
from django.shortcuts import get_object_or_404, render
from django.template.loader import render_to_string
from django.urls import reverse
from django.http import HttpResponse
from .models import (
LegislativeText,
LegislationBook,
LegislationClassification,
School,
Country,
Sponsor,
Category,
models_in_index,
)
from random import sample
def legislation_to_html(legislation):
return render_to_string("explorer/comp_legislation.html", {
"legislation": legislation,
})
def index(request):
legislative_texts = list(LegislativeText.objects.all())
try:
legislative_texts = sample(legislative_texts, 5)
except ValueError:
# there's not enough texts, so just return nothing
legislative_texts = []
legislative_texts = [
legislation_to_html(text) for text in legislative_texts
]
context = {
"legislation": legislative_texts,
}
return render(request, "explorer/index.html", context)
def all(request):
legislative_texts = list(LegislativeText.objects.all())
context = {
"legislative_texts": legislative_texts,
}
return render(request, "explorer/all.html", context)
def view_legislation(request, legislation_id):
legislation = get_object_or_404(LegislativeText, pk=legislation_id)
context = {
"legislation": legislation,
"lines": legislation.get_lines()
}
return render(request, "explorer/legislation.html", context)
def view_conference(request, conference_id):
book = get_object_or_404(LegislationBook, pk=conference_id)
results = LegislativeText.objects.filter(from_book=book)
context = {
"book": book,
"legislation": results,
"sample": results[0]
}
return render(request, "explorer/conference.html", context)
def stats(request):
all_legislation = len(LegislativeText.objects.all())
context = {
"all": all_legislation,
"red_senate": len(LegislativeText.objects.filter(assembly="RSB")),
"blue_senate": len(LegislativeText.objects.filter(assembly="BSB")),
"white_senate": len(LegislativeText.objects.filter(assembly="WSB")),
"red_house": len(LegislativeText.objects.filter(assembly="RHB")),
"blue_house": len(LegislativeText.objects.filter(assembly="BHB")),
"white_house": len(LegislativeText.objects.filter(assembly="WHB")),
"red_ga": len(LegislativeText.objects.filter(assembly="RGA")),
"blue_ga": len(LegislativeText.objects.filter(assembly="BGA")),
"white_ga": len(LegislativeText.objects.filter(assembly="WGA")),
}
return render(request, "explorer/stats.html", context)
def get_all_classified_by_id(request, model_id):
classification = get_object_or_404(LegislationClassification, pk=model_id)
# this is very expensive; make a way for this to be cached please?
all_texts = LegislativeText.objects.all()
all_terms = classification.text_to_match.split(',')
all_terms = [i.lower() for i in all_terms]
matches = []
for text in all_texts:
for term in all_terms:
if term in text.text.lower():
matches.append(text)
break
return render(request, "explorer/results.html", {
"legislation": matches,
"result_name": "All legislation in topic {}".format(classification.name)
})
def get_all_by_x(model):
def wrapped(request, model_id):
instance = get_object_or_404(model, pk=model_id)
legislation = instance.legislativetext_set.all()
legislation = [legislation_to_html(i) for i in legislation]
return render(request, "explorer/results.html", {
"result_name": "All legislation by {}".format(instance.name),
"legislation": legislation
})
return wrapped
def get_all_xs(model):
def wrapper(request):
instances = model.objects.all()
try:
# what the heck, django?????
plural = model._meta.verbose_name_plural
except:
plural = model.__name__ + "s"
plural = plural.lower()
return render(request, "explorer/listing.html", {
"result_name": "All {}".format(plural),
"instances": instances,
})
return wrapper
def return_groups(request):
listing = {}
for model in models_in_index:
try:
name = model._meta.verbose_name.lower()
except:
name = model.__name__.lower()
listing[name] = reverse(model.__name__)
print(listing)
return render(request, "explorer/by_group.html", { "listing": listing })
def handle_search(request):
try:
query = request.GET['search_term']
except KeyError:
return render(request, "explorer/search.html", {})
f = LegislativeText.objects.filter
text_results = f(text__icontains=query)
title_results = f(legislation_title__icontains=query)
school_results = f(school__name__icontains=query)
sponsor_results = f(sponsors__name__icontains=query)
country_results = f(country__name__icontains=query)
results = text_results.union(
title_results,
school_results,
sponsor_results,
country_results
)
results = [legislation_to_html(i) for i in results]
return render(request, "explorer/results.html", {
"result_name": "Results for search term '{}'".format(query),
"legislation": results
})
get_all_by_school = get_all_by_x(School)
get_all_by_country = get_all_by_x(Country)
get_all_by_sponsor = get_all_by_x(Sponsor)
get_all_by_category = get_all_by_x(Category)
get_all_by_conference = get_all_by_x(LegislationBook)
get_all_schools = get_all_xs(School)
get_all_countries = get_all_xs(Country)
get_all_sponsors = get_all_xs(Sponsor)
get_all_categories = get_all_xs(Category)
get_all_classifications = get_all_xs(LegislationClassification)
get_all_conferences = get_all_xs(LegislationBook)

View File

@ -1,16 +0,0 @@
"""
ASGI config for franklincce project.
It exposes the ASGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/4.2/howto/deployment/asgi/
"""
import os
from django.core.asgi import get_asgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'franklincce.settings')
application = get_asgi_application()

View File

@ -1,132 +0,0 @@
"""
Django settings for franklincce project.
Generated by 'django-admin startproject' using Django 4.2.12.
For more information on this file, see
https://docs.djangoproject.com/en/4.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.2/ref/settings/
"""
import os
import random
from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
SECRET_KEY = os.environ.get("SECRET_KEY")
if not SECRET_KEY:
print("[!!!] YOU'RE USING A RANDOM SECRET_KEY -- CHANGE THIS IF YOU'RE GOING INTO PROD")
SECRET_KEY = random.randint(1, 100000000000000)
DEBUG = bool(os.environ.get("DEBUG", default=0))
# 'DJANGO_ALLOWED_HOSTS' should be a single string of hosts with a space between each.
# For example: 'DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]'
try:
ALLOWED_HOSTS = os.environ.get("DJANGO_ALLOWED_HOSTS").split(" ")
except:
ALLOWED_HOSTS = ["127.0.0.1", "localhost"]
INSTALLED_APPS = [
'explorer.apps.ExplorerConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'franklincce.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'franklincce.wsgi.application'
# Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
STATICFILES_DIRS = [
BASE_DIR / "static",
]
# Password validation
# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/4.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.2/howto/static-files/
STATIC_URL = '/static/'
# Default primary key field type
# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field
STATIC_ROOT = BASE_DIR / "staticfiles"
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
CSRF_TRUSTED_ORIGINS = ["http://localhost:1337", "https://franklincce.beepboop.systems"]

View File

@ -1,25 +0,0 @@
"""
URL configuration for franklincce project.
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/4.2/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import include, path
from django.conf import settings
from django.conf.urls.static import static
urlpatterns = [
path('explorer/', include("explorer.urls")),
path('admin/', admin.site.urls),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

View File

@ -1,16 +0,0 @@
"""
WSGI config for franklincce project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/4.2/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'franklincce.settings')
application = get_wsgi_application()

View File

@ -1,22 +0,0 @@
#!/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/bin/python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'franklincce.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()

View File

@ -1,3 +0,0 @@
django==4.2.12
pymupdf==1.23.26
gunicorn

View File

@ -1,123 +0,0 @@
:root {
--component-bg-color: rgba(10, 10, 10, 0.7);
--text-color: rgb(150, 150, 150);
--accented-text-color: rgb(255, 255, 255);
}
body {
font-family: sans-serif;
}
a {
color: lightblue;
}
a:active {
color: orangered;
}
a:visited {
color: mediumpurple;
}
::-moz-selection ::selection {
color: blue;
}
nav#navbar {
display: flex; /* make it from left to right */
flex-direction: row;
background-color: var(--component-bg-color);
background-clip: padding-box;
padding-left: 10px;
padding-right: 10px;
}
#leftnav {
padding-right: 10px;
}
#rightnav {
padding-left: 10px;
/* make this part right justified */
margin-left: auto;
margin-right: 0;
}
#legcontainer {
display: flex;
flex-direction: row;
gap: 10px;
}
#leginfo {
width: 15em;
}
#legislation {
width: 100%;
}
h1, h2, h3, h4, h5, h6 {
color: var(--accented-text-color);
word-wrap: break-word;
}
p, td, i, ul, li {
color: var(--text-color);
font-size: 16px;
}
.no-margin-top {
margin-top: 0px;
}
.boxed {
margin-top: 10px;
padding: 5px;
padding-top: 0px;
border: solid 1px;
background-color: var(--component-bg-color);
border-radius: 15px;
}
.legnumbers {
text-align: right;
user-select: none;
}
@media only screen and (max-width: 280px) {
#rightnav {
display: none;
}
}
@media only screen and (max-width: 570px) {
nav#navbar {
padding-bottom: 10px;
}
#legcontainer {
flex-direction: column;
padding-right: 10px;
}
#leginfo {
width: 100%;
}
}
@media (pointer: coarse) {
#legcontainer {
flex-direction: column;
padding-right: 10px;
}
#leginfo {
width: 100%;
}
p, td, i, ul, li {
font-size: 26px;
}
nav#navbar {
font-size: 40px;
margin-bottom: 10px;
}
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 149 KiB

View File

@ -1,7 +0,0 @@
body {
background-image: url("/static/tn-cap.jpg");
background-position: center;
background-attachment: fixed;
background-repeat: no-repeat;
background-size: cover;
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.8 MiB

View File

@ -1,7 +0,0 @@
body {
background-image: url("/static/un-gen-asm.jpg");
background-position: center;
background-attachment: fixed;
background-repeat: no-repeat;
background-size: cover;
}

View File

@ -1,13 +0,0 @@
files=$(find kb -type f | awk -F'/' '{print $NF}')
IFS='
'
mkdir -p franklincce/staticfiles/kb
mkdir -p franklincce/staticfiles/root
for file in $files; do
without_extension=${file%.*}
echo $file, $without_extension
pandoc -s --template=./template.html -f markdown -t html -o "franklincce/staticfiles/kb/$without_extension.html" "kb/$without_extension.md" --lua-filter=links-to-html.lua
done
cp franklincce/staticfiles/kb/web_root.html franklincce/staticfiles/root/index.html

View File

@ -1,16 +0,0 @@
---
title: "Contributing to the knowledgebase"
---
1. If you haven't a GitHub account, [create one](https://github.com/signup) and sign in.
2. Navigate to the `yig` repository in the VSCode Web Editor, or click [here](https://github.dev/stupidcomputer/yig)
3. Click on the `kb/` directory on the sidebar.
4. Select an article that you'd like to change.
5. Make your changes.
6. Click on the 'source control' tab on the left (the one that looks like a tree)
7. Click the "plus" icon on all the entries in the left panel, and write a message explaining your changes, and then click "Commit and Push"
8. You'll get a prompt to create your own "fork"; say yes.
9. Navigate back to [Github](https://github.com) and click on the "yig" link in the upper left area.
10. Click the "Compare and pull request" button, then scroll down and click "create pull request".
Thanks for your contribution!

View File

@ -1,4 +0,0 @@
# franklincce kb
This directory contains the sources for the knowledge-base portion of the Franklin CCE Delegation's website.
It includes bill and resolution writing tips and guides.
If you want more information on contributing to the knowledge-base but you don't really know your way around git, see [here](./CONTRIBUTING.md).

View File

@ -1,33 +0,0 @@
---
title: "Choosing a resolution topic"
---
This is a tricky thing.
You've already picked a country, and so you have to choose a topic to write your resolution on.
Here are some things to keep in mind as you're choosing a topic.
1. Choose a *worldwide* topic.
Unless you're requesting immediate aid in the wake of a disaster, personalized solutions (ie. solutions only applicable to one country) are not the point of the UN.
Make the solutions enumerated in your resolution applicable to other countries, too.
2. Choose a *memorable* topic.
Don't do a resolution about drilling wells in Africa for water, because that's been done to death.
Try to come up with a unique way to solve the problems at hand.
One of the components which your resolutions will be scored on is originality, so keep that in mind.
3. Choose a topic with an asymmetrical *information advantage*.
What is a topic with an "asymmetrical information advantage"?
It's a topic that is common and prevalant, but not a lot of people know the intricate details of.
Like the American healthcare system, or the ICAO.
This is less important than the other ones, but is still something to consider -- choose a topic with an information advantage and it makes it easier during debate.
4. Choose an *important* topic.
This is the most important one.
Choose a relevant topic.
If you're country's in the news recently, look there for inspiration.
Got a good idea? Great. Now it's time to [write your resolution](./writing-resolution.md).

View File

@ -1,10 +0,0 @@
---
title: "Franklin CCE Knowledgebase"
---
## Model UN
- [Writing a resolution](./writing-resolution.md)
Want to see more articles in the knowledgebase?
[Learn how to contribute](./CONTRIBUTING.md).

View File

@ -1,38 +0,0 @@
---
title: "Operative verbs reference"
---
- Accepts
- Adopts
- Agrees
- Appeals
- Approves
- Authorizes
- Calls upon
- Commends
- Considers
- Decides
- Declares
- Determines
- Directs
- Emphasizes
- Encourages
- Endorses
- Expresses appreciation
- Expresses hope
- Invites
- Notes
- Notes with approval
- Notes with concern
- Notes with satisfaction
- Proclaims
- Reaffirms
- Recommends
- Reminds
- Repeals
- Requests
- Resolves
- Suggests
- Supports
- Takes note
- Urges

View File

@ -1,40 +0,0 @@
---
title: "Preambular verbs reference"
---
- Acknowledging
- Affirming
- Appreciating
- Approving
- Aware
- Bearing in mind
- Believing
- Commending
- Concerned
- Conscious
- Considering
- Convinced
- Desiring
- Emphasizing
- Expecting
- Expressing
- Fully aware
- Guided by
- Having adopted
- Having considered
- Having noted
- Having reviewed
- Mindful
- Noting
- Noting with approval
- Noting with concern
- Noting with satisfaction
- Observing
- Realising
- Recalling
- Recognizing
- Seeking
- Taking into consideration
- Underlining
- Welcoming
- Whereas

View File

@ -1,6 +0,0 @@
---
title: "Franklin CCE Delegation website"
---
This website contains some materials for the Franklin High School YMCA Center for Civic Engagement delegation.
Specifically, a [bill database](/explorer) and a [collection of helpful articles](/knowledge) on various things are available here.

View File

@ -1,67 +0,0 @@
---
title: "Writing a Resolution"
---
*Note: This guide assumes you've already choosen a topic; if not, see [Choosing a resolution topic](./choosing-topic-resolution.md).*
The General Assembly, the largest law-making body of the UN, expresses its intentions and actions as a *resolution*.
In this article, we'll learn how to write one.
A resolution is composed of two parts: the *preambulatory* (or preambular) and *operative* clauses.
Preambulatory clauses provide background on why your resolution is needed, the problem it addresses, and sets the tone for the operative clauses.
Operative clauses are the items of action; they are directives that countries agree to.
(Only the Security Council has the power of passing *mandatory* directives.)
As an example, let's assume we're writing a resolution that condemns kicking puppies. (I would hope that you'd agree with the premise; if not -- that's an interesting opinion.) There was a recent puppy kicking event in the country of `Madeupia`, so let's note that:
> Noting with concern the International Puppy Kicking Event held in `Madeupia` on 22 September 2024, where various kinds of dogs were subject to unconsionable acts of depravity and abuse;
This is a good example of a preambulatory clause -- it states background for the operative clauses. If someone asked you if people actually kick dogs, this is your response. It demonstrates the necessity of the resolution.
There also might be past resulutions about animal abuse or dogs, so it might be prudent to mention those, too:
> Affirming resolution UNEP/5/1, which specifically emphasizes the value of animals and the contributions their welfare makes to the UN Sustainable Development Goals,
You essentially want to create many of these preambulatory clauses to paint a vivid picture of the need of your resolution. If you want to see more verbs to go in front of your preambulatory clauses, check out the [preambulatory verbs reference](./preambular-verbs.md).
Once you're done writing your preambulatory clauses, you write:
> The General Assembly hereby:
You do **NOT** write:
> The Delegation of (your country here) hereby:
because the resolution is adopted by the Assembly. That's why there's a vote and stuff -- if a country wants to express a feeling it feels by itself, it can submit a Communique to the UN instead.
Now it's time to talk about operative clauses.
These clauses are where the actual works gets done.
I think a good first move is the condemn the country of `Madeupia` for their puppy kicking event:
> Condemns, in the strongest terms, the affront to animal welfare perpetuated by the State of Madeupia;
Here are some other forms of clauses you might want to consider:
> Takes note of the report of the Secretary-General on "Animal Welfare as it pertains to the advancement of the SDGs";
>
> Expresses support of the Totally Real Animal Anti-Violence Directorate of the United Nations and their actions in response to this issue;
>
> Recognizes the positive contribution of various Member States towards the resolution of this issue, including the Republic of Dog Lovers;
Do note -- when you're ending your resolution, your final clause ends with a period.
> Requests the Secretary-General to submit a report on the implementation of the present resolution including recommendations for future action at the next session of the General Assembly.
For a full list of operative verbs, see the [operative verbs reference](./operative-verbs.md).
One final note: if you use a comma in any of your clauses, all of the clauses (except the last one) have to end in a semicolon.
This is because of the rules of Standard English.
## Some other resources for resolution writing
- [The UN's Model UN resources](https://www.un.org/en/model-united-nations/drafting-resolutions)
The UN does say some different things in this article -- it talks about L-documents and agreement annotations and soliciting agreement from other delegations beforehand.
That's not relevant to our conference, but it is still a good resource.
For an example of a finished resolution that is somewhat up to format, see the heading "Compilation Text as of 21 October 2016 (Rev. 3) The General Assembly".

View File

@ -1,4 +0,0 @@
function Link(el)
el.target = string.gsub(el.target, "%.md", ".html")
return el
end

View File

@ -1,4 +0,0 @@
FROM nginx:1.25
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d

View File

@ -1,39 +0,0 @@
upstream franklincce {
server web:8000;
}
server {
listen 80;
server_name franklincce.beepboop.systems;
port_in_redirect off;
absolute_redirect off;
server_name_in_redirect off;
location /explorer/ {
proxy_pass http://franklincce;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
client_max_body_size 100M;
}
location /admin/ {
proxy_pass http://franklincce;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
client_max_body_size 100M;
}
location /static/ {
alias /home/app/web/staticfiles/;
}
location /knowledge {
alias /home/app/web/staticfiles/kb;
}
location / {
alias /home/app/web/staticfiles/root/;
}
}

View File

@ -1,4 +1,9 @@
{ pkgs ? import <nixpkgs> {} }: { pkgs ? import <nixpkgs> {} }:
pkgs.mkShell { pkgs.mkShell {
nativeBuildInputs = with pkgs.python311Packages; [ django pymupdf ] ++ [ pkgs.docker-compose pkgs.gnumake pkgs.pandoc ] ; # nativeBuildInputs is usually what you want -- tools you need to run
} nativeBuildInputs = with pkgs; [
buildPackages.python311Packages.pymupdf
buildPackages.python311Packages.flask
buildPackages.python311Packages.flask-bootstrap
];
}

View File

@ -1,27 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>$title$</title>
<link rel="stylesheet" href="/static/style.css">
<link rel="stylesheet" href="/static/tn.css">
</head>
<body>
<nav id="navbar" class="boxed">
<div id="leftnav">
<a href="/explorer">explorer</a>
<a href="/knowledge">knowledge</a>
</div>
</nav>
<div class="boxed">
<h1>$title$</h1>
$body$
</div>
</body>
</html>