From 8ead20a7a457e778fa0507ba016200f83e07d4c2 Mon Sep 17 00:00:00 2001 From: Jonathan Yu Date: Tue, 9 Jul 2019 11:10:25 -0400 Subject: [PATCH 01/24] Updated FDNY agency instructions --- data/agencies.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data/agencies.json b/data/agencies.json index a83de6af3..18e82e206 100644 --- a/data/agencies.json +++ b/data/agencies.json @@ -588,7 +588,7 @@ }, "monitor_agency_requests": [], "specific_request_instructions": { - "text": "

The Fire Department of the City of New York commonly issues records to the public. Below are instructions for obtaining common documents by mail or in-person or through Open Records, along with forms that are needed to complete the process. Note, all mail requests must include a check or money order for applicable fees payable to \"NYC Fire Department\" (no cash accepted). Include a stamped, self-addressed envelope. For in-person record requests please visit:

FDNY Public Records Unit
9 MetroTech Center - First Floor
Brooklyn, N.Y. 11201
Use the FLATBUSH AVENUE ENTRANCE
Hours of Operation:
Monday - Friday 9:00am to 4:00pm (except Holidays)

Please note the following records that you can obtain in person through the Public Records Unit:

Please note that all records provided through Open Records/FOIL will be redacted. If you require records with no redactions, then you must serve a “So Ordered” subpoena. For further instructions please refer to https://www1.nyc.gov/site/fdny/about/resources/record-requests/records-request.page.

Building Related Reports

These records can be obtained by Mail, In Person or through Open Records (see dropdown below for instructions)

Copy of a Violation

Violation Special Report

Fuel Tank Special Report/Environmental Assessment Report

Fire Reports

Fire Incident Report

These records can be obtained by Mail, In Person or through Open Records (see dropdown below for instructions). The Fire Incent Report must be obtained prior to requesting the Fire Marshal Investigation Report to make a determination if there is a Fire Marshal Investigation Report.

Fire Marshall Investigation Report

The following record request can only be handled by mail.

Emergency Related Report (FIRE/EMS)

Ambulance Call Report/ Pre-Hospital Care Report

These records can be obtained by Mail, In Person or Online through https://fdny.mypatientencounters.com/myrecord

911 Call Report (CAD) (NOT Ambulance Call Report/ Pre-Hospital Care Report)

These records can be obtained by Mail, In Person or through OpenRecords (see dropdown below for instructions).

If your request does not fall under any of these categories please select “Other Request” below.

" + "text": "

The Fire Department of the City of New York commonly provides access to the records to the public consistent with Federal, State and NYC laws. Below are instructions for obtaining commonly requested records by Mail, In Person or Online through Open Records, along with forms that are needed to complete the process. Note all mail requests must include a check or money order for applicable fees payable to 'NYC Fire Department' (no cash accepted) and a stamped, self-addressed envelope. For in person record requests please visit:

FDNY Public Records Unit
9 MetroTech Center - First Floor
Brooklyn, N.Y. 11201
Use the FLATBUSH AVENUE ENTRANCE
Hours of Operation:
Monday - Friday 9:00am to 4:00pm (except Holidays)

Records provided through Open Records/FOIL will be redacted. If you require records with no redactions, then you must serve a 'So Ordered' subpoena. For further instructions please refer to https://www1.nyc.gov/site/fdny/about/resources/record-requests/records-request.page.

Please note the following records that you can obtain in person through the Public Records Unit:

Building Related Reports

These records can be obtained by Mail, In Person or through Open Records (see dropdown below for instructions)

Copy of a Violation

Violation Special Report

Fuel Tank Special Report/Environmental Assessment Report

Fire Reports

Fire Incident Report
These records can be obtained by Mail, In Person or Online through Open Records (see dropdown below for instructions). The Fire Incent Report must be obtained prior to requesting the Fire Marshal Investigation Report to make a determination if there is a Fire Marshal Investigation Report.

Fire Marshal Investigation Report The following record request can only be handled by mail.

Emergency Related Reports (FIRE/EMS)

These records can be obtained by Mail, In Person or Online through https://fdny.mypatientencounters.com/myrecord. Please note ACR records cannot be obtained through Open Records.

911 Call Report (CAD) (NOT Ambulance Call Report/ Pre-Hospital Care Report)
These records can be obtained by Mail, In Person or Online through Open Records (see dropdown below for instructions).

If your request does not fall under any of these categories please select 'Other Request' below

" } }, "acronym": "FDNY" From 715c55557b83708c242079f46d917bd5657c72fe Mon Sep 17 00:00:00 2001 From: Joel Castillo Date: Wed, 14 Aug 2019 12:14:32 -0400 Subject: [PATCH 02/24] Delay display of title to allow for PII review. The title may be displayed in the following circumstances: - The currently logged in user is in the requests assigned agency users OR - The current User is the requester OR - The title is not private AND - The request was acknowledged OR - The request was denied OR - The request is overdue for an acknowledgment In order to reduce code duplication, a new method has been added to the Requests model (`show_title`) which returns a bool to determine if the title should be displayed on the frontend. This function is used in request/views.py and in search/utils.py --- .startup/fakesmtp_startup.sh | 0 .startup/flask_startup.sh | 0 .startup/tmux_setup.sh | 0 app/models.py | 1440 +++++++++++++++++++--------------- app/request/views.py | 335 ++++---- app/search/utils.py | 531 +++++++------ 6 files changed, 1263 insertions(+), 1043 deletions(-) mode change 100644 => 100755 .startup/fakesmtp_startup.sh mode change 100644 => 100755 .startup/flask_startup.sh mode change 100644 => 100755 .startup/tmux_setup.sh diff --git a/.startup/fakesmtp_startup.sh b/.startup/fakesmtp_startup.sh old mode 100644 new mode 100755 diff --git a/.startup/flask_startup.sh b/.startup/flask_startup.sh old mode 100644 new mode 100755 diff --git a/.startup/tmux_setup.sh b/.startup/tmux_setup.sh old mode 100644 new mode 100755 diff --git a/app/models.py b/app/models.py index 4ded338eb..ed15d7d95 100644 --- a/app/models.py +++ b/app/models.py @@ -9,27 +9,16 @@ from elasticsearch.helpers import bulk from flask import current_app, session -from flask_login import ( - UserMixin, - AnonymousUserMixin -) +from flask_login import UserMixin, AnonymousUserMixin, current_user from functools import reduce from operator import ior from sqlalchemy import desc -from sqlalchemy.dialects.postgresql import ( - ARRAY, - JSONB -) +from sqlalchemy.dialects.postgresql import ARRAY, JSONB from sqlalchemy.orm import column_property from sqlalchemy.orm.exc import MultipleResultsFound from warnings import warn -from app import ( - db, - es, - calendar, - sentry -) +from app import db, es, calendar, sentry from app.constants import ( ES_DATETIME_FORMAT, permission, @@ -40,7 +29,7 @@ determination_type, response_privacy, submission_methods, - event_type + event_type, ) from app.constants.request_date import RELEASE_PUBLIC_DAYS from app.constants.schemas import AGENCIES_SCHEMA @@ -48,7 +37,7 @@ from app.lib.utils import ( eval_request_bool, DuplicateFileException, - InvalidDeterminationException + InvalidDeterminationException, ) @@ -64,7 +53,8 @@ class Roles(db.Model): permissions -- Column: Integer users -- Relationship: 'User', 'role' """ - __tablename__ = 'roles' + + __tablename__ = "roles" id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(64), unique=True) permissions = db.Column(db.BigInteger) @@ -75,85 +65,81 @@ def populate(cls): Insert permissions for each role. """ roles = { - role_name.ANONYMOUS: ( - permission.NONE - ), - role_name.PUBLIC_REQUESTER: ( - permission.ADD_NOTE - ), + role_name.ANONYMOUS: (permission.NONE), + role_name.PUBLIC_REQUESTER: (permission.ADD_NOTE), role_name.AGENCY_HELPER: ( - permission.ADD_NOTE | - permission.ADD_FILE | - permission.ADD_LINK | - permission.ADD_OFFLINE_INSTRUCTIONS + permission.ADD_NOTE + | permission.ADD_FILE + | permission.ADD_LINK + | permission.ADD_OFFLINE_INSTRUCTIONS ), role_name.AGENCY_OFFICER: ( - permission.ACKNOWLEDGE | - permission.DENY | - permission.EXTEND | - permission.CLOSE | - permission.RE_OPEN | - permission.ADD_NOTE | - permission.ADD_FILE | - permission.ADD_LINK | - permission.ADD_OFFLINE_INSTRUCTIONS | - permission.GENERATE_LETTER | - permission.EDIT_NOTE | - permission.EDIT_NOTE_PRIVACY | - permission.EDIT_FILE | - permission.EDIT_FILE_PRIVACY | - permission.EDIT_LINK | - permission.EDIT_LINK_PRIVACY | - permission.EDIT_OFFLINE_INSTRUCTIONS | - permission.EDIT_OFFLINE_INSTRUCTIONS_PRIVACY | - permission.EDIT_OFFLINE_INSTRUCTIONS | - permission.EDIT_FILE_PRIVACY | - permission.DELETE_NOTE | - permission.DELETE_FILE | - permission.DELETE_LINK | - permission.DELETE_OFFLINE_INSTRUCTIONS | - permission.EDIT_TITLE | - permission.CHANGE_PRIVACY_TITLE | - permission.EDIT_AGENCY_REQUEST_SUMMARY | - permission.CHANGE_PRIVACY_AGENCY_REQUEST_SUMMARY | - permission.EDIT_REQUESTER_INFO + permission.ACKNOWLEDGE + | permission.DENY + | permission.EXTEND + | permission.CLOSE + | permission.RE_OPEN + | permission.ADD_NOTE + | permission.ADD_FILE + | permission.ADD_LINK + | permission.ADD_OFFLINE_INSTRUCTIONS + | permission.GENERATE_LETTER + | permission.EDIT_NOTE + | permission.EDIT_NOTE_PRIVACY + | permission.EDIT_FILE + | permission.EDIT_FILE_PRIVACY + | permission.EDIT_LINK + | permission.EDIT_LINK_PRIVACY + | permission.EDIT_OFFLINE_INSTRUCTIONS + | permission.EDIT_OFFLINE_INSTRUCTIONS_PRIVACY + | permission.EDIT_OFFLINE_INSTRUCTIONS + | permission.EDIT_FILE_PRIVACY + | permission.DELETE_NOTE + | permission.DELETE_FILE + | permission.DELETE_LINK + | permission.DELETE_OFFLINE_INSTRUCTIONS + | permission.EDIT_TITLE + | permission.CHANGE_PRIVACY_TITLE + | permission.EDIT_AGENCY_REQUEST_SUMMARY + | permission.CHANGE_PRIVACY_AGENCY_REQUEST_SUMMARY + | permission.EDIT_REQUESTER_INFO ), role_name.AGENCY_ADMIN: ( - permission.ACKNOWLEDGE | - permission.DENY | - permission.EXTEND | - permission.CLOSE | - permission.RE_OPEN | - permission.ADD_NOTE | - permission.ADD_FILE | - permission.ADD_LINK | - permission.ADD_OFFLINE_INSTRUCTIONS | - permission.GENERATE_LETTER | - permission.EDIT_NOTE | - permission.EDIT_NOTE_PRIVACY | - permission.EDIT_FILE | - permission.EDIT_FILE_PRIVACY | - permission.EDIT_LINK | - permission.EDIT_LINK_PRIVACY | - permission.EDIT_OFFLINE_INSTRUCTIONS | - permission.EDIT_OFFLINE_INSTRUCTIONS_PRIVACY | - permission.EDIT_FILE_PRIVACY | - permission.EDIT_TITLE | - permission.DELETE_NOTE | - permission.DELETE_FILE | - permission.DELETE_LINK | - permission.DELETE_OFFLINE_INSTRUCTIONS | - permission.CHANGE_PRIVACY_TITLE | - permission.EDIT_AGENCY_REQUEST_SUMMARY | - permission.CHANGE_PRIVACY_AGENCY_REQUEST_SUMMARY | - permission.ADD_USER_TO_REQUEST | - permission.REMOVE_USER_FROM_REQUEST | - permission.EDIT_USER_REQUEST_PERMISSIONS | - permission.ADD_USER_TO_AGENCY | - permission.REMOVE_USER_FROM_AGENCY | - permission.CHANGE_USER_ADMIN_PRIVILEGE | - permission.EDIT_REQUESTER_INFO - ) + permission.ACKNOWLEDGE + | permission.DENY + | permission.EXTEND + | permission.CLOSE + | permission.RE_OPEN + | permission.ADD_NOTE + | permission.ADD_FILE + | permission.ADD_LINK + | permission.ADD_OFFLINE_INSTRUCTIONS + | permission.GENERATE_LETTER + | permission.EDIT_NOTE + | permission.EDIT_NOTE_PRIVACY + | permission.EDIT_FILE + | permission.EDIT_FILE_PRIVACY + | permission.EDIT_LINK + | permission.EDIT_LINK_PRIVACY + | permission.EDIT_OFFLINE_INSTRUCTIONS + | permission.EDIT_OFFLINE_INSTRUCTIONS_PRIVACY + | permission.EDIT_FILE_PRIVACY + | permission.EDIT_TITLE + | permission.DELETE_NOTE + | permission.DELETE_FILE + | permission.DELETE_LINK + | permission.DELETE_OFFLINE_INSTRUCTIONS + | permission.CHANGE_PRIVACY_TITLE + | permission.EDIT_AGENCY_REQUEST_SUMMARY + | permission.CHANGE_PRIVACY_AGENCY_REQUEST_SUMMARY + | permission.ADD_USER_TO_REQUEST + | permission.REMOVE_USER_FROM_REQUEST + | permission.EDIT_USER_REQUEST_PERMISSIONS + | permission.ADD_USER_TO_AGENCY + | permission.REMOVE_USER_FROM_AGENCY + | permission.CHANGE_USER_ADMIN_PRIVILEGE + | permission.EDIT_REQUESTER_INFO + ), } for name, value in roles.items(): @@ -165,7 +151,7 @@ def populate(cls): db.session.commit() def __repr__(self): - return '' % self.name + return "" % self.name class Agencies(db.Model): @@ -193,13 +179,16 @@ class Agencies(db.Model): OpenRecords """ - __tablename__ = 'agencies' + + __tablename__ = "agencies" ein = db.Column(db.String(4), primary_key=True) parent_ein = db.Column(db.String(3)) categories = db.Column(ARRAY(db.String(256))) - _name = db.Column(db.String(256), nullable=False, name='name') + _name = db.Column(db.String(256), nullable=False, name="name") acronym = db.Column(db.String(64), nullable=True) - _next_request_number = db.Column(db.Integer(), db.Sequence('request_seq'), name='next_request_number') + _next_request_number = db.Column( + db.Integer(), db.Sequence("request_seq"), name="next_request_number" + ) default_email = db.Column(db.String(254)) appeals_email = db.Column(db.String(254)) is_active = db.Column(db.Boolean(), default=False) @@ -208,34 +197,34 @@ class Agencies(db.Model): # TODO: Use validation on agency_features column administrators = db.relationship( - 'Users', + "Users", secondary="agency_users", primaryjoin="and_(Agencies.ein == AgencyUsers.agency_ein, " - "AgencyUsers.is_agency_active == True, " - "AgencyUsers.is_agency_admin == True)", - secondaryjoin="AgencyUsers.user_guid == Users.guid" + "AgencyUsers.is_agency_active == True, " + "AgencyUsers.is_agency_admin == True)", + secondaryjoin="AgencyUsers.user_guid == Users.guid", ) standard_users = db.relationship( - 'Users', + "Users", secondary="agency_users", primaryjoin="and_(Agencies.ein == AgencyUsers.agency_ein, " - "AgencyUsers.is_agency_active == True, " - "AgencyUsers.is_agency_admin == False)", - secondaryjoin="AgencyUsers.user_guid == Users.guid" + "AgencyUsers.is_agency_active == True, " + "AgencyUsers.is_agency_admin == False)", + secondaryjoin="AgencyUsers.user_guid == Users.guid", ) active_users = db.relationship( - 'Users', + "Users", secondary="agency_users", primaryjoin="and_(Agencies.ein == AgencyUsers.agency_ein, " - "AgencyUsers.is_agency_active == True)", - secondaryjoin="AgencyUsers.user_guid == Users.guid" + "AgencyUsers.is_agency_active == True)", + secondaryjoin="AgencyUsers.user_guid == Users.guid", ) inactive_users = db.relationship( - 'Users', + "Users", secondary="agency_users", primaryjoin="and_(Agencies.ein == AgencyUsers.agency_ein, " - "AgencyUsers.is_agency_active == False)", - secondaryjoin="AgencyUsers.user_guid == Users.guid" + "AgencyUsers.is_agency_active == False)", + secondaryjoin="AgencyUsers.user_guid == Users.guid", ) @property @@ -255,11 +244,12 @@ def parent(self): @property def next_request_number(self): from app.lib.db_utils import update_object + num = self._next_request_number update_object( - {'_next_request_number': self._next_request_number + 1}, + {"_next_request_number": self._next_request_number + 1}, Agencies, - self.formatted_parent_ein + self.formatted_parent_ein, ) return num @@ -269,8 +259,11 @@ def next_request_number(self, value): @property def name(self): - return '{name} ({acronym})'.format(name=self._name, acronym=self.acronym) if self.acronym else '{name}'.format( - name=self._name) + return ( + "{name} ({acronym})".format(name=self._name, acronym=self.acronym) + if self.acronym + else "{name}".format(name=self._name) + ) @name.setter def name(self, value): @@ -281,35 +274,43 @@ def populate(cls, json_name=None): """ Automatically populate the agencies table for the OpenRecords application. """ - filename = json_name or current_app.config['AGENCY_DATA'] - with open(filename, 'r') as data: + filename = json_name or current_app.config["AGENCY_DATA"] + with open(filename, "r") as data: data = json.load(data) if not validate_schema(data, AGENCIES_SCHEMA): - warn("Invalid JSON Data. Not importing any agencies.", category=UserWarning) + warn( + "Invalid JSON Data. Not importing any agencies.", + category=UserWarning, + ) return False - for agency in data['agencies']: - if Agencies.query.filter_by(ein=agency['ein']).first() is not None: - warn("Duplicate EIN ({ein}); Row not imported".format(ein=agency['ein']), category=UserWarning) + for agency in data["agencies"]: + if Agencies.query.filter_by(ein=agency["ein"]).first() is not None: + warn( + "Duplicate EIN ({ein}); Row not imported".format( + ein=agency["ein"] + ), + category=UserWarning, + ) continue a = cls( - ein=agency['ein'], - parent_ein=agency['parent_ein'], - categories=agency['categories'], - name=agency['name'], - acronym=agency.get('acronym', None), - next_request_number=agency['next_request_number'], - default_email=agency['default_email'], - appeals_email=agency['appeals_email'], - is_active=agency['is_active'], - agency_features=agency['agency_features'] + ein=agency["ein"], + parent_ein=agency["parent_ein"], + categories=agency["categories"], + name=agency["name"], + acronym=agency.get("acronym", None), + next_request_number=agency["next_request_number"], + default_email=agency["default_email"], + appeals_email=agency["appeals_email"], + is_active=agency["is_active"], + agency_features=agency["agency_features"], ) db.session.add(a) db.session.commit() def __repr__(self): - return '' % self.name + return "" % self.name class Users(UserMixin, db.Model): @@ -334,7 +335,8 @@ class Users(UserMixin, db.Model): fax_number - string containing the user's fax number mailing_address - a JSON object containing the user's address """ - __tablename__ = 'users' + + __tablename__ = "users" guid = db.Column(db.String(64), unique=True, primary_key=True) is_nyc_employee = db.Column(db.Boolean, default=False) has_nyc_account = db.Column(db.Boolean, default=False) @@ -352,23 +354,24 @@ class Users(UserMixin, db.Model): organization = db.Column(db.String(128)) # Outside organization phone_number = db.Column(db.String(25)) fax_number = db.Column(db.String(25)) - _mailing_address = db.Column(JSONB, - name='mailing_address') # TODO: define validation for minimum acceptable mailing address + _mailing_address = db.Column( + JSONB, name="mailing_address" + ) # TODO: define validation for minimum acceptable mailing address session_id = db.Column(db.String(254), nullable=True, default=None) signature = db.Column(db.String(), nullable=True, default=None) fullname = column_property(first_name + " " + last_name) # Relationships - user_requests = db.relationship("UserRequests", backref="user", lazy='dynamic') + user_requests = db.relationship("UserRequests", backref="user", lazy="dynamic") agencies = db.relationship( - 'Agencies', + "Agencies", secondary="agency_users", primaryjoin="AgencyUsers.user_guid == Users.guid", secondaryjoin="and_(AgencyUsers.agency_ein == Agencies.ein, " - "AgencyUsers.is_agency_active == True)", - lazy='dynamic' + "AgencyUsers.is_agency_active == True)", + lazy="dynamic", ) - agency_users = db.relationship("AgencyUsers", backref="user", lazy='dynamic') + agency_users = db.relationship("AgencyUsers", backref="user", lazy="dynamic") @property def is_authenticated(self): @@ -376,12 +379,12 @@ def is_authenticated(self): Verifies the access token currently stored in the user's session by invoking the OAuth User Web Service and checking the response. """ - if current_app.config['USE_LDAP']: + if current_app.config["USE_LDAP"]: return True - if current_app.config['USE_SAML']: - if session.get('samlUserdata', None): + if current_app.config["USE_SAML"]: + if session.get("samlUserdata", None): return True - if current_app.config['USE_LOCAL_AUTH']: + if current_app.config["USE_LOCAL_AUTH"]: return True return False @@ -420,8 +423,14 @@ def default_agency_ein(self): Return the Users default agency ein. :return: String """ - agency = AgencyUsers.query.join(Users).filter(AgencyUsers.is_primary_agency == True, - AgencyUsers.user_guid == self.guid).one_or_none() + agency = ( + AgencyUsers.query.join(Users) + .filter( + AgencyUsers.is_primary_agency == True, + AgencyUsers.user_guid == self.guid, + ) + .one_or_none() + ) if agency is not None: return agency.agency_ein return None @@ -462,7 +471,9 @@ def anonymous_request(self): if this user is an anonymous requester. """ if self.is_anonymous_requester: - return Requests.query.filter_by(id=self.user_requests.one().request_id).one() + return Requests.query.filter_by( + id=self.user_requests.one().request_id + ).one() return None @property @@ -515,12 +526,17 @@ def is_agency_active(self, ein=None): def agencies_for_forms(self): agencies = self.agencies.with_entities(Agencies.ein, Agencies._name).all() - agencies.insert(0, agencies.pop(agencies.index((self.default_agency.ein, self.default_agency._name)))) + agencies.insert( + 0, + agencies.pop( + agencies.index((self.default_agency.ein, self.default_agency._name)) + ), + ) return agencies @property def name(self): - return ' '.join((self.first_name.title(), self.last_name.title())) + return " ".join((self.first_name.title(), self.last_name.title())) @property def mailing_address(self): @@ -534,8 +550,8 @@ def mailing_address(self, mailing_address): def formatted_point_of_contact_number(self): if self.phone_number: formatted_phone_number = self.phone_number - formatted_phone_number = formatted_phone_number.strip('(') - formatted_phone_number = formatted_phone_number.replace(') ', '-') + formatted_phone_number = formatted_phone_number.strip("(") + formatted_phone_number = formatted_phone_number.replace(") ", "-") return formatted_phone_number def get_id(self): @@ -546,23 +562,23 @@ def es_update(self): Call es_update for any request where this user is the requester since the request es doc relies on the requester's name. """ - if current_app.config['ELASTICSEARCH_ENABLED']: + if current_app.config["ELASTICSEARCH_ENABLED"]: requests = [request.id for request in self.requests] - actions = [{ - '_op_type': 'update', - '_id': request_id, - 'doc': { - 'requester_id': self.guid, - 'requester_name': self.name + actions = [ + { + "_op_type": "update", + "_id": request_id, + "doc": {"requester_id": self.guid, "requester_name": self.name}, } - } for request_id in requests] + for request_id in requests + ] bulk( es, actions, - index=current_app.config['ELASTICSEARCH_INDEX'], - doc_type='request', - chunk_size=current_app.config['ELASTICSEARCH_CHUNK_SIZE'] + index=current_app.config["ELASTICSEARCH_INDEX"], + doc_type="request", + chunk_size=current_app.config["ELASTICSEARCH_CHUNK_SIZE"], ) @property @@ -591,35 +607,35 @@ def val_for_events(self): @classmethod def populate(cls, csv_name=None): - filename = csv_name or current_app.config['STAFF_DATA'] - with open(filename, 'r') as data: + filename = csv_name or current_app.config["STAFF_DATA"] + with open(filename, "r") as data: dictreader = csv.DictReader(data) for row in dictreader: - if Users.query.filter_by(email=row['email']).first() is None: + if Users.query.filter_by(email=row["email"]).first() is None: user = cls( guid=str(uuid4()), - is_super=eval(row['is_super']), - first_name=row['first_name'], - middle_initial=row['middle_initial'], - last_name=row['last_name'], - email=row['email'], - email_validated=eval(row['email_validated']), - terms_of_use_accepted=eval(row['terms_of_use_accepted']), - phone_number=row['phone_number'], - fax_number=row['fax_number'] + is_super=eval(row["is_super"]), + first_name=row["first_name"], + middle_initial=row["middle_initial"], + last_name=row["last_name"], + email=row["email"], + email_validated=eval(row["email_validated"]), + terms_of_use_accepted=eval(row["terms_of_use_accepted"]), + phone_number=row["phone_number"], + fax_number=row["fax_number"], ) db.session.add(user) db.session.commit() - agency_eins = row['agencies'].split('|') + agency_eins = row["agencies"].split("|") for agency in agency_eins: - ein, is_active, is_admin, is_primary_agency = agency.split('#') + ein, is_active, is_admin, is_primary_agency = agency.split("#") agency_user = AgencyUsers( user_guid=user.guid, agency_ein=ein, is_agency_active=eval_request_bool(is_active), is_agency_admin=eval_request_bool(is_admin), - is_primary_agency=eval_request_bool(is_primary_agency) + is_primary_agency=eval_request_bool(is_primary_agency), ) db.session.add(agency_user) if agency_eins: @@ -631,7 +647,7 @@ def __init__(self, **kwargs): super(Users, self).__init__(**kwargs) def __repr__(self): - return ''.format(self.get_id()) + return "".format(self.get_id()) class Anonymous(AnonymousUserMixin): @@ -669,7 +685,7 @@ def is_agency(self): return False def __repr__(self): - return '' + return "" class AgencyUsers(db.Model): @@ -685,19 +701,18 @@ class AgencyUsers(db.Model): primary_agency - a boolean value that determines whether the agency identified by agency_ein is the users default agency """ - __tablename__ = 'agency_users' + + __tablename__ = "agency_users" user_guid = db.Column(db.String(64), db.ForeignKey("users.guid"), primary_key=True) - agency_ein = db.Column(db.String(4), db.ForeignKey("agencies.ein"), primary_key=True) + agency_ein = db.Column( + db.String(4), db.ForeignKey("agencies.ein"), primary_key=True + ) is_agency_active = db.Column(db.Boolean, default=False, nullable=False) is_agency_admin = db.Column(db.Boolean, default=False, nullable=False) is_primary_agency = db.Column(db.Boolean, default=False, nullable=False) __table_args__ = ( - db.ForeignKeyConstraint( - [user_guid], - [Users.guid], - onupdate="CASCADE" - ), + db.ForeignKeyConstraint([user_guid], [Users.guid], onupdate="CASCADE"), ) @@ -723,79 +738,92 @@ class Requests(db.Model): agency_request_summary_release_date - a datetime of when the agency_request_summary will be made public custom_metadata - a JSON that contains the metadata from an agency's custom request forms """ - __tablename__ = 'requests' + + __tablename__ = "requests" id = db.Column(db.String(19), primary_key=True) - agency_ein = db.Column(db.String(4), db.ForeignKey('agencies.ein')) - category = db.Column(db.String, default='All', - nullable=False) # FIXME: should be nullable, 'All' shouldn't be used + agency_ein = db.Column(db.String(4), db.ForeignKey("agencies.ein")) + category = db.Column( + db.String, default="All", nullable=False + ) # FIXME: should be nullable, 'All' shouldn't be used title = db.Column(db.String(90)) description = db.Column(db.String(5000)) date_created = db.Column(db.DateTime, default=datetime.utcnow()) - date_submitted = db.Column(db.DateTime) # used to calculate due date, rounded off to next business day + date_submitted = db.Column( + db.DateTime + ) # used to calculate due date, rounded off to next business day date_closed = db.Column(db.DateTime, default=None, nullable=True) due_date = db.Column(db.DateTime) submission = db.Column( - db.Enum(submission_methods.DIRECT_INPUT, - submission_methods.FAX, - submission_methods.PHONE, - submission_methods.EMAIL, - submission_methods.MAIL, - submission_methods.IN_PERSON, - submission_methods.THREE_ONE_ONE, - name='submission')) + db.Enum( + submission_methods.DIRECT_INPUT, + submission_methods.FAX, + submission_methods.PHONE, + submission_methods.EMAIL, + submission_methods.MAIL, + submission_methods.IN_PERSON, + submission_methods.THREE_ONE_ONE, + name="submission", + ) + ) status = db.Column( - db.Enum(request_status.OPEN, - request_status.IN_PROGRESS, - request_status.DUE_SOON, # within the next 5 business days - request_status.OVERDUE, - request_status.CLOSED, - name='status'), - nullable=False + db.Enum( + request_status.OPEN, + request_status.IN_PROGRESS, + request_status.DUE_SOON, # within the next 5 business days + request_status.OVERDUE, + request_status.CLOSED, + name="status", + ), + nullable=False, ) privacy = db.Column(JSONB) agency_request_summary = db.Column(db.String(5000)) agency_request_summary_release_date = db.Column(db.DateTime) custom_metadata = db.Column(JSONB) - user_requests = db.relationship('UserRequests', backref=db.backref('request', uselist=False), lazy='dynamic') - agency = db.relationship('Agencies', backref='requests', uselist=False) - responses = db.relationship('Responses', backref=db.backref('request', uselist=False), lazy='dynamic') + user_requests = db.relationship( + "UserRequests", backref=db.backref("request", uselist=False), lazy="dynamic" + ) + agency = db.relationship("Agencies", backref="requests", uselist=False) + responses = db.relationship( + "Responses", backref=db.backref("request", uselist=False), lazy="dynamic" + ) requester = db.relationship( - 'Users', - secondary='user_requests', # expects table name + "Users", + secondary="user_requests", # expects table name primaryjoin=lambda: Requests.id == UserRequests.request_id, secondaryjoin="and_(Users.guid == UserRequests.user_guid, " - "UserRequests.request_user_type == '{}')".format(user_type_request.REQUESTER), + "UserRequests.request_user_type == '{}')".format(user_type_request.REQUESTER), backref="requests", viewonly=True, - uselist=False + uselist=False, ) # any agency user associated with a request is considered an assigned user agency_users = db.relationship( - 'Users', - secondary='user_requests', + "Users", + secondary="user_requests", primaryjoin=lambda: Requests.id == UserRequests.request_id, secondaryjoin="and_(Users.guid == UserRequests.user_guid, " - "UserRequests.request_user_type == '{}')".format(user_type_request.AGENCY), - viewonly=True + "UserRequests.request_user_type == '{}')".format(user_type_request.AGENCY), + viewonly=True, ) - PRIVACY_DEFAULT = {'title': False, 'agency_request_summary': True} + PRIVACY_DEFAULT = {"title": False, "agency_request_summary": True} def __init__( - self, - id, - title, - description, - agency_ein, - date_created, - category=None, - privacy=None, - date_submitted=None, # FIXME: are some of these really nullable? - due_date=None, - submission=None, - status=request_status.OPEN, - custom_metadata=None + self, + id, + title, + description, + agency_ein, + date_created, + category=None, + privacy=None, + date_submitted=None, # FIXME: are some of these really nullable? + due_date=None, + submission=None, + status=request_status.OPEN, + custom_metadata=None, ): self.id = id self.title = title @@ -819,34 +847,81 @@ def val_for_events(self): be the same on Request creation are not included. """ return { - 'title': self.title, - 'description': self.description, - 'due_date': self.due_date.isoformat(), + "title": self.title, + "description": self.description, + "due_date": self.due_date.isoformat(), } @property def was_acknowledged(self): - if self.responses.join(Determinations).filter( - Determinations.dtype == determination_type.ACKNOWLEDGMENT).one_or_none() is not None: + if ( + self.responses.join(Determinations) + .filter(Determinations.dtype == determination_type.ACKNOWLEDGMENT) + .one_or_none() + is not None + ): return True return False @property def was_reopened(self): - return self.responses.join(Determinations).filter( - Determinations.dtype == determination_type.REOPENING).first() is not None + return ( + self.responses.join(Determinations) + .filter(Determinations.dtype == determination_type.REOPENING) + .first() + is not None + ) @property def last_date_closed(self): if self.status == request_status.CLOSED: - return self.responses.join(Determinations).filter( - Determinations.dtype.in_([determination_type.CLOSING, determination_type.DENIAL]) - ).order_by(desc(Determinations.date_modified)).limit(1).one().date_modified + return ( + self.responses.join(Determinations) + .filter( + Determinations.dtype.in_( + [determination_type.CLOSING, determination_type.DENIAL] + ) + ) + .order_by(desc(Determinations.date_modified)) + .limit(1) + .one() + .date_modified + ) return None @property def days_until_due(self): - return calendar.busdaycount(datetime.utcnow(), self.due_date.replace(hour=23, minute=59, second=59)) + return calendar.busdaycount( + datetime.utcnow(), self.due_date.replace(hour=23, minute=59, second=59) + ) + + @property + def show_title(self) -> bool: + """Determine whether the title should be displayed on the front-end. + + The title may be displayed in the following circumstances: + - The currently logged in user is in the requests assigned agency users OR + - The current User is the requester OR + - The title is not private AND + - The request was acknowledged OR + - The request was denied OR + - The request is overdue for an acknowledgment + + Returns: + bool: True if the title should be shown, False otherwise. + """ + return ( + current_user in self.agency_users + or current_user == self.requester + or ( + not self.privacy["title"] + and ( + self.was_acknowledged + or self.last_date_closed is not None + or self.days_until_due < 0 + ) + ) + ) @property def url(self): @@ -856,42 +931,53 @@ def url(self): Since we cannot use SERVER_NAME in config (and, by extension, 'url_for'), BASE_URL and VIEW_REQUEST_ENDPOINT will have to do. """ - return urljoin(current_app.config['BASE_URL'], - "{view_request_endpoint}/{request_id}".format( - view_request_endpoint=current_app.config['VIEW_REQUEST_ENDPOINT'], - request_id=self.id - )) + return urljoin( + current_app.config["BASE_URL"], + "{view_request_endpoint}/{request_id}".format( + view_request_endpoint=current_app.config["VIEW_REQUEST_ENDPOINT"], + request_id=self.id, + ), + ) @property def agency_request_summary_released(self): """ Determine whether the agency_request_summary has been made public and has passed its release date """ - return self.status == request_status.CLOSED and not self.privacy['agency_request_summary'] and \ - self.agency_request_summary and self.agency_request_summary_release_date is not None and \ - self.agency_request_summary_release_date < datetime.utcnow() + return ( + self.status == request_status.CLOSED + and not self.privacy["agency_request_summary"] + and self.agency_request_summary + and self.agency_request_summary_release_date is not None + and self.agency_request_summary_release_date < datetime.utcnow() + ) def es_update(self): - if current_app.config['ELASTICSEARCH_ENABLED']: + if current_app.config["ELASTICSEARCH_ENABLED"]: if self.agency.is_active: es.update( index=current_app.config["ELASTICSEARCH_INDEX"], - doc_type='request', + doc_type="request", id=self.id, body={ - 'doc': { - 'title': self.title, - 'description': self.description, - 'agency_request_summary': self.agency_request_summary, - 'assigned_users': [user.get_id() for user in self.agency_users], - 'title_private': self.privacy['title'], - 'agency_request_summary_private': not self.agency_request_summary_released, - 'date_due': self.due_date.strftime(ES_DATETIME_FORMAT), - 'date_closed': self.date_closed.strftime( - ES_DATETIME_FORMAT) if self.date_closed is not None else [], - 'status': self.status, - 'requester_name': self.requester.name, - 'public_title': 'Private' if self.privacy['title'] else self.title + "doc": { + "title": self.title, + "description": self.description, + "agency_request_summary": self.agency_request_summary, + "assigned_users": [ + user.get_id() for user in self.agency_users + ], + "title_private": self.privacy["title"], + "agency_request_summary_private": not self.agency_request_summary_released, + "date_due": self.due_date.strftime(ES_DATETIME_FORMAT), + "date_closed": self.date_closed.strftime(ES_DATETIME_FORMAT) + if self.date_closed is not None + else [], + "status": self.status, + "requester_name": self.requester.name, + "public_title": "Private" + if self.privacy["title"] + else self.title, } }, # refresh='wait_for' @@ -899,48 +985,51 @@ def es_update(self): def es_create(self): """ Must be called AFTER UserRequest has been created. """ - if current_app.config['ELASTICSEARCH_ENABLED']: + if current_app.config["ELASTICSEARCH_ENABLED"]: + print("Private" if not self.show_title else self.title) es.create( index=current_app.config["ELASTICSEARCH_INDEX"], - doc_type='request', + doc_type="request", id=self.id, body={ - 'title': self.title, - 'description': self.description, - 'agency_request_summary': self.agency_request_summary, - 'agency_ein': self.agency_ein, - 'agency_name': self.agency.name, - 'assigned_users': [user.get_id() for user in self.agency_users], - 'agency_acronym': self.agency.acronym, - 'title_private': self.privacy['title'], - 'agency_request_summary_private': not self.agency_request_summary_released, - 'date_created': self.date_created.strftime(ES_DATETIME_FORMAT), - 'date_submitted': self.date_submitted.strftime(ES_DATETIME_FORMAT), - 'date_received': self.date_created.strftime( - ES_DATETIME_FORMAT) if self.date_created < self.date_submitted else self.date_submitted.strftime( - ES_DATETIME_FORMAT), - 'date_due': self.due_date.strftime(ES_DATETIME_FORMAT), - 'submission': self.submission, - 'status': self.status, - 'requester_id': (self.requester.get_id() - if not self.requester.is_anonymous_requester - else ''), - 'requester_name': self.requester.name, - 'public_title': 'Private' if self.privacy['title'] else self.title, - } + "title": self.title, + "description": self.description, + "agency_request_summary": self.agency_request_summary, + "agency_ein": self.agency_ein, + "agency_name": self.agency.name, + "assigned_users": [user.get_id() for user in self.agency_users], + "agency_acronym": self.agency.acronym, + "title_private": self.privacy["title"], + "agency_request_summary_private": not self.agency_request_summary_released, + "date_created": self.date_created.strftime(ES_DATETIME_FORMAT), + "date_submitted": self.date_submitted.strftime(ES_DATETIME_FORMAT), + "date_received": self.date_created.strftime(ES_DATETIME_FORMAT) + if self.date_created < self.date_submitted + else self.date_submitted.strftime(ES_DATETIME_FORMAT), + "date_due": self.due_date.strftime(ES_DATETIME_FORMAT), + "submission": self.submission, + "status": self.status, + "requester_id": ( + self.requester.get_id() + if not self.requester.is_anonymous_requester + else "" + ), + "requester_name": self.requester.name, + "public_title": "Private" if not self.show_title else self.title, + }, ) def es_delete(self): """ Delete a document from the elastic search index """ - if current_app.config['ELASTICSEARCH_ENABLED']: + if current_app.config["ELASTICSEARCH_ENABLED"]: es.delete( index=current_app.config["ELASTICSEARCH_INDEX"], - doc_type='request', - id=self.id + doc_type="request", + id=self.id, ) def __repr__(self): - return '' % self.id + return "" % self.id class Events(db.Model): @@ -957,40 +1046,37 @@ class Events(db.Model): previous_value - a string containing the old value of the event new_value - a string containing the new value of the event """ - __tablename__ = 'events' + + __tablename__ = "events" id = db.Column(db.Integer, primary_key=True) - request_id = db.Column(db.String(19), db.ForeignKey('requests.id')) + request_id = db.Column(db.String(19), db.ForeignKey("requests.id")) user_guid = db.Column(db.String(64)) - response_id = db.Column(db.Integer, db.ForeignKey('responses.id')) + response_id = db.Column(db.Integer, db.ForeignKey("responses.id")) type = db.Column(db.String(64)) timestamp = db.Column(db.DateTime, default=datetime.utcnow()) previous_value = db.Column(JSONB) new_value = db.Column(JSONB) __table_args__ = ( - db.ForeignKeyConstraint( - [user_guid], - [Users.guid], - onupdate="CASCADE" - ), + db.ForeignKeyConstraint([user_guid], [Users.guid], onupdate="CASCADE"), ) response = db.relationship("Responses", backref="events") request = db.relationship("Requests", backref="events") user = db.relationship( - "Users", - primaryjoin="Events.user_guid == Users.guid", - backref="events" + "Users", primaryjoin="Events.user_guid == Users.guid", backref="events" ) - def __init__(self, - request_id, - user_guid, - type_, - previous_value=None, - new_value=None, - response_id=None, - timestamp=None): + def __init__( + self, + request_id, + user_guid, + type_, + previous_value=None, + new_value=None, + response_id=None, + timestamp=None, + ): self.request_id = request_id self.user_guid = user_guid self.response_id = response_id @@ -1000,18 +1086,17 @@ def __init__(self, self.timestamp = timestamp or datetime.utcnow() def __repr__(self): - return '' % self.id + return "" % self.id @property def affected_user(self): if self.new_value is not None and "user_guid" in self.new_value: - return Users.query.filter_by( - guid=self.new_value["user_guid"] - ).one() + return Users.query.filter_by(guid=self.new_value["user_guid"]).one() class RowContent(object): - - def __init__(self, event, verb, string, affected_user=None, no_user_string=None): + def __init__( + self, event, verb, string, affected_user=None, no_user_string=None + ): """ :param verb: action describing event :param string: format()-ready string where first field is verb and @@ -1032,7 +1117,7 @@ def __str__(self): if self.no_user_string is not None and self.event.user is None: string = self.no_user_string else: - string = ' '.join((self.event.user.name, self.string)) + string = " ".join((self.event.user.name, self.string)) return string.format(*format_args) @staticmethod @@ -1047,67 +1132,96 @@ def history_row_content(self): """ if self.type == event_type.REQ_STATUS_CHANGED: return "This request's status was changed to:
{}".format( - self.new_value['status']) + self.new_value["status"] + ) valid_types = { - event_type.USER_ADDED: - self.RowContent(self, "added", "{} user: {}.", self.affected_user, "User {}: {}."), - event_type.USER_REMOVED: - self.RowContent(self, "removed", "{} user: {}.", self.affected_user), - event_type.USER_PERM_CHANGED: - self.RowContent(self, "changed", "{} permssions for user: {}.", self.affected_user), - event_type.REQUESTER_INFO_EDITED: - self.RowContent(self, "changed", "{} the requester's information."), - event_type.REQ_CREATED: - self.RowContent(self, "created", "{} this request."), - event_type.AGENCY_REQ_CREATED: - self.RowContent(self, "created", "{} this request on behalf of {}.", self.request.requester), - event_type.REQ_ACKNOWLEDGED: - self.RowContent(self, "acknowledged", "{} this request."), - event_type.REQ_EXTENDED: - self.RowContent(self, "extended", "{} this request."), - event_type.REQ_CLOSED: - self.RowContent(self, "closed", "{} this request."), - event_type.REQ_DENIED: - self.RowContent(self, "denied", "{} this request."), - event_type.REQ_REOPENED: - self.RowContent(self, "re-opened", "{} this request."), - event_type.REQ_TITLE_EDITED: - self.RowContent(self, "changed", "{} the title."), - event_type.REQ_AGENCY_REQ_SUM_EDITED: - self.RowContent(self, "changed", "{} the agency request summary."), - event_type.REQ_TITLE_PRIVACY_EDITED: - self.RowContent(self, "changed", "{} the title privacy."), - event_type.REQ_AGENCY_REQ_SUM_PRIVACY_EDITED: - self.RowContent(self, "changed", "{} the agency request summary privacy."), - event_type.FILE_ADDED: - self.RowContent(self, "added", "{} a file response."), - event_type.FILE_EDITED: - self.RowContent(self, "changed", "{} a file response."), - event_type.FILE_REMOVED: - self.RowContent(self, "deleted", "{} a file response."), - event_type.LINK_ADDED: - self.RowContent(self, "added", "{} a link response."), - event_type.LINK_EDITED: - self.RowContent(self, "changed", "{} a link response."), - event_type.LINK_REMOVED: - self.RowContent(self, "deleted", "{} a link response."), - event_type.INSTRUCTIONS_ADDED: - self.RowContent(self, "added", "{} an offline instructions response."), - event_type.INSTRUCTIONS_EDITED: - self.RowContent(self, "changed", "{} an offline instructions response."), - event_type.INSTRUCTIONS_REMOVED: - self.RowContent(self, "deleted", "{} an offline instructions response."), - event_type.NOTE_ADDED: - self.RowContent(self, "added", "{} a note response."), - event_type.NOTE_EDITED: - self.RowContent(self, "changed", "{} a note response."), - event_type.NOTE_REMOVED: - self.RowContent(self, "deleted", "{} a note response."), - event_type.ENVELOPE_CREATED: - self.RowContent(self, "created", "{} an envelope."), - event_type.RESPONSE_LETTER_CREATED: - self.RowContent(self, "added", "{} a letter.") + event_type.USER_ADDED: self.RowContent( + self, "added", "{} user: {}.", self.affected_user, "User {}: {}." + ), + event_type.USER_REMOVED: self.RowContent( + self, "removed", "{} user: {}.", self.affected_user + ), + event_type.USER_PERM_CHANGED: self.RowContent( + self, "changed", "{} permssions for user: {}.", self.affected_user + ), + event_type.REQUESTER_INFO_EDITED: self.RowContent( + self, "changed", "{} the requester's information." + ), + event_type.REQ_CREATED: self.RowContent( + self, "created", "{} this request." + ), + event_type.AGENCY_REQ_CREATED: self.RowContent( + self, + "created", + "{} this request on behalf of {}.", + self.request.requester, + ), + event_type.REQ_ACKNOWLEDGED: self.RowContent( + self, "acknowledged", "{} this request." + ), + event_type.REQ_EXTENDED: self.RowContent( + self, "extended", "{} this request." + ), + event_type.REQ_CLOSED: self.RowContent(self, "closed", "{} this request."), + event_type.REQ_DENIED: self.RowContent(self, "denied", "{} this request."), + event_type.REQ_REOPENED: self.RowContent( + self, "re-opened", "{} this request." + ), + event_type.REQ_TITLE_EDITED: self.RowContent( + self, "changed", "{} the title." + ), + event_type.REQ_AGENCY_REQ_SUM_EDITED: self.RowContent( + self, "changed", "{} the agency request summary." + ), + event_type.REQ_TITLE_PRIVACY_EDITED: self.RowContent( + self, "changed", "{} the title privacy." + ), + event_type.REQ_AGENCY_REQ_SUM_PRIVACY_EDITED: self.RowContent( + self, "changed", "{} the agency request summary privacy." + ), + event_type.FILE_ADDED: self.RowContent( + self, "added", "{} a file response." + ), + event_type.FILE_EDITED: self.RowContent( + self, "changed", "{} a file response." + ), + event_type.FILE_REMOVED: self.RowContent( + self, "deleted", "{} a file response." + ), + event_type.LINK_ADDED: self.RowContent( + self, "added", "{} a link response." + ), + event_type.LINK_EDITED: self.RowContent( + self, "changed", "{} a link response." + ), + event_type.LINK_REMOVED: self.RowContent( + self, "deleted", "{} a link response." + ), + event_type.INSTRUCTIONS_ADDED: self.RowContent( + self, "added", "{} an offline instructions response." + ), + event_type.INSTRUCTIONS_EDITED: self.RowContent( + self, "changed", "{} an offline instructions response." + ), + event_type.INSTRUCTIONS_REMOVED: self.RowContent( + self, "deleted", "{} an offline instructions response." + ), + event_type.NOTE_ADDED: self.RowContent( + self, "added", "{} a note response." + ), + event_type.NOTE_EDITED: self.RowContent( + self, "changed", "{} a note response." + ), + event_type.NOTE_REMOVED: self.RowContent( + self, "deleted", "{} a note response." + ), + event_type.ENVELOPE_CREATED: self.RowContent( + self, "created", "{} an envelope." + ), + event_type.RESPONSE_LETTER_CREATED: self.RowContent( + self, "added", "{} a letter." + ), } if self.type in valid_types: @@ -1125,45 +1239,49 @@ class Responses(db.Model): content - a JSON object that contains the content for all the possible responses a request can have privacy - an Enum containing the privacy options for a response """ - __tablename__ = 'responses' + + __tablename__ = "responses" id = db.Column(db.Integer, primary_key=True) - request_id = db.Column(db.String(19), db.ForeignKey('requests.id')) - privacy = db.Column(db.Enum( - response_privacy.PRIVATE, - response_privacy.RELEASE_AND_PRIVATE, - response_privacy.RELEASE_AND_PUBLIC, - name="privacy")) + request_id = db.Column(db.String(19), db.ForeignKey("requests.id")) + privacy = db.Column( + db.Enum( + response_privacy.PRIVATE, + response_privacy.RELEASE_AND_PRIVATE, + response_privacy.RELEASE_AND_PUBLIC, + name="privacy", + ) + ) date_modified = db.Column(db.DateTime) release_date = db.Column(db.DateTime) deleted = db.Column(db.Boolean, default=False, nullable=False) is_editable = db.Column(db.Boolean, default=False, nullable=False) - type = db.Column(db.Enum( - response_type.NOTE, - response_type.LINK, - response_type.FILE, - response_type.INSTRUCTIONS, - response_type.DETERMINATION, - response_type.EMAIL, - response_type.LETTER, - response_type.ENVELOPE, - name='type' - )) - - __mapper_args__ = {'polymorphic_on': type} + type = db.Column( + db.Enum( + response_type.NOTE, + response_type.LINK, + response_type.FILE, + response_type.INSTRUCTIONS, + response_type.DETERMINATION, + response_type.EMAIL, + response_type.LETTER, + response_type.ENVELOPE, + name="type", + ) + ) + + __mapper_args__ = {"polymorphic_on": type} # TODO: overwrite filter to automatically check if deleted=False - def __init__(self, - request_id, - privacy, - date_modified=None, - is_editable=False): + def __init__(self, request_id, privacy, date_modified=None, is_editable=False): self.request_id = request_id self.privacy = privacy self.date_modified = date_modified or datetime.utcnow() - self.release_date = (calendar.addbusdays(datetime.utcnow(), RELEASE_PUBLIC_DAYS) - if privacy == response_privacy.RELEASE_AND_PUBLIC - else None) + self.release_date = ( + calendar.addbusdays(datetime.utcnow(), RELEASE_PUBLIC_DAYS) + if privacy == response_privacy.RELEASE_AND_PUBLIC + else None + ) self.is_editable = is_editable # NOTE: If you can find a way to make this class work with abc, @@ -1176,24 +1294,29 @@ def preview(self): @property def val_for_events(self): """ JSON to store in Events 'new_value' field. """ - val = { - c.name: getattr(self, c.name) - for c in self.__table__.columns - } - val.pop('id') - val['privacy'] = self.privacy + val = {c.name: getattr(self, c.name) for c in self.__table__.columns} + val.pop("id") + val["privacy"] = self.privacy return val @property def is_public(self): - return (self.privacy == response_privacy.RELEASE_AND_PUBLIC and - self.release_date is not None and - datetime.utcnow() > self.release_date) + return ( + self.privacy == response_privacy.RELEASE_AND_PUBLIC + and self.release_date is not None + and datetime.utcnow() > self.release_date + ) @property def creator(self): - return Events.query.filter(Events.response_id == self.id, - Events.type.in_(event_type.RESPONSE_ADDED_TYPES)).one().user + return ( + Events.query.filter( + Events.response_id == self.id, + Events.type.in_(event_type.RESPONSE_ADDED_TYPES), + ) + .one() + .user + ) @property def communication_method_type(self): @@ -1203,9 +1326,14 @@ def communication_method_type(self): :return: response_type.LETTER or response_type.EMAIL :rtype: str """ - communication_methods = CommunicationMethods.query.filter_by(response_id=self.id).all() - return response_type.LETTER if response_type.LETTER in [cm.method_type for cm in - communication_methods] else response_type.EMAIL + communication_methods = CommunicationMethods.query.filter_by( + response_id=self.id + ).all() + return ( + response_type.LETTER + if response_type.LETTER in [cm.method_type for cm in communication_methods] + else response_type.EMAIL + ) @property def event_timestamp(self): @@ -1214,7 +1342,11 @@ def event_timestamp(self): the newest timestamp of the newest event which will be displayed on the frontend. :return: timestamp of the newest event row associated with a response """ - timestamps = Events.query.filter_by(response_id=self.id).order_by(desc(Events.timestamp)).all() + timestamps = ( + Events.query.filter_by(response_id=self.id) + .order_by(desc(Events.timestamp)) + .all() + ) if timestamps: return timestamps[0].timestamp return self.date_modified @@ -1225,7 +1357,7 @@ def make_public(self): db.session.commit() def __repr__(self): - return '' % self.id + return "" % self.id class Reasons(db.Model): @@ -1241,35 +1373,40 @@ class Reasons(db.Model): Reason are based off the Law Department's responses. """ - __tablename__ = 'reasons' + + __tablename__ = "reasons" id = db.Column(db.Integer, primary_key=True) - type = db.Column(db.Enum( - determination_type.CLOSING, - determination_type.DENIAL, - determination_type.REOPENING, - name="reason_type" - ), nullable=False) - agency_ein = db.Column(db.String(4), db.ForeignKey('agencies.ein')) + type = db.Column( + db.Enum( + determination_type.CLOSING, + determination_type.DENIAL, + determination_type.REOPENING, + name="reason_type", + ), + nullable=False, + ) + agency_ein = db.Column(db.String(4), db.ForeignKey("agencies.ein")) title = db.Column(db.String, nullable=False) content = db.Column(db.String, nullable=False) has_appeals_language = db.Column(db.Boolean, default=True) @classmethod def populate(cls): - with open(current_app.config['REASON_DATA'], 'r') as data: + with open(current_app.config["REASON_DATA"], "r") as data: dictreader = csv.DictReader(data) for row in dictreader: - agency_ein = row['agency_ein'] if row['agency_ein'] else None + agency_ein = row["agency_ein"] if row["agency_ein"] else None reason = cls( - type=row['type'], - title=row['title'], - content=row['content'], - has_appeals_language=eval_request_bool(row['has_appeals_language']), - agency_ein=agency_ein + type=row["type"], + title=row["title"], + content=row["content"], + has_appeals_language=eval_request_bool(row["has_appeals_language"]), + agency_ein=agency_ein, ) - if not Reasons.query.filter_by(title=row['title'], content=row['content'], - agency_ein=agency_ein).first(): + if not Reasons.query.filter_by( + title=row["title"], content=row["content"], agency_ein=agency_ein + ).first(): db.session.add(reason) db.session.commit() @@ -1289,13 +1426,19 @@ class UserRequests(db.Model): for public information. point_of_contact = a boolean to determine the point of contact of a request """ - __tablename__ = 'user_requests' + + __tablename__ = "user_requests" user_guid = db.Column(db.String(64), primary_key=True) - request_id = db.Column(db.String(19), db.ForeignKey("requests.id"), primary_key=True) + request_id = db.Column( + db.String(19), db.ForeignKey("requests.id"), primary_key=True + ) request_user_type = db.Column( - db.Enum(user_type_request.REQUESTER, - user_type_request.AGENCY, - name='request_user_type')) + db.Enum( + user_type_request.REQUESTER, + user_type_request.AGENCY, + name="request_user_type", + ) + ) permissions = db.Column(db.BigInteger) point_of_contact = db.Column(db.Boolean, default=False) # Note: If an anonymous user creates a request, they will be listed in the UserRequests table, but will have the @@ -1303,11 +1446,7 @@ class UserRequests(db.Model): # current anonymous user is in fact the requester. __table_args__ = ( - db.ForeignKeyConstraint( - [user_guid], - [Users.guid], - onupdate="CASCADE" - ), + db.ForeignKeyConstraint([user_guid], [Users.guid], onupdate="CASCADE"), ) @property @@ -1319,7 +1458,7 @@ def val_for_events(self): "user_guid": self.user_guid, "request_user_type": self.request_user_type, "permissions": self.permissions, - "point_of_contact": self.point_of_contact + "point_of_contact": self.point_of_contact, } def has_permission(self, perm): @@ -1355,7 +1494,9 @@ def set_permissions(self, permissions): db.session.commit() def get_permission_choice_indices(self): - return [i for i, p in enumerate(permission.ALL) if bool(self.permissions & p.value)] + return [ + i for i, p in enumerate(permission.ALL) if bool(self.permissions & p.value) + ] class ResponseTokens(db.Model): @@ -1367,15 +1508,15 @@ class ResponseTokens(db.Model): response_id - a foreign key that links to a response's primary key expiration_date - a datetime object containing the date at which this token becomes invalid """ - __tablename__ = 'response_tokens' + + __tablename__ = "response_tokens" id = db.Column(db.Integer, primary_key=True) token = db.Column(db.String, nullable=False) response_id = db.Column(db.Integer, db.ForeignKey("responses.id"), nullable=False) response = db.relationship("Responses", backref=db.backref("token", uselist=False)) - def __init__(self, - response_id): + def __init__(self, response_id): self.token = self.generate_token() self.response_id = response_id @@ -1391,21 +1532,16 @@ class Notes(Responses): id - an integer that is the primary key of Notes content - a string that contains the content of a note """ + __tablename__ = response_type.NOTE - __mapper_args__ = {'polymorphic_identity': response_type.NOTE} + __mapper_args__ = {"polymorphic_identity": response_type.NOTE} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) content = db.Column(db.String(5000)) - def __init__(self, - request_id, - privacy, - content, - date_modified=None, - is_editable=False): - super(Notes, self).__init__(request_id, - privacy, - date_modified, - is_editable) + def __init__( + self, request_id, privacy, content, date_modified=None, is_editable=False + ): + super(Notes, self).__init__(request_id, privacy, date_modified, is_editable) self.content = content @property @@ -1424,8 +1560,9 @@ class Files(Responses): size - a string containing the size of a file hash - a string containing the sha1 hash of a file """ + __tablename__ = response_type.FILE - __mapper_args__ = {'polymorphic_identity': response_type.FILE} + __mapper_args__ = {"polymorphic_identity": response_type.FILE} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) title = db.Column(db.String(250)) name = db.Column(db.String) @@ -1433,41 +1570,31 @@ class Files(Responses): size = db.Column(db.Integer) hash = db.Column(db.String) - def __init__(self, - request_id, - privacy, - title, - name, - mime_type, - size, - hash_, - date_modified=None, - is_editable=False): + def __init__( + self, + request_id, + privacy, + title, + name, + mime_type, + size, + hash_, + date_modified=None, + is_editable=False, + ): try: file_exists = Files.query.filter_by(request_id=request_id, hash=hash_).all() for file in file_exists: if not file.deleted: - raise DuplicateFileException( - file_name=name, - request_id=request_id - ) + raise DuplicateFileException(file_name=name, request_id=request_id) except DuplicateFileException: sentry.captureException() - raise DuplicateFileException( - file_name=name, - request_id=request_id - ) + raise DuplicateFileException(file_name=name, request_id=request_id) except MultipleResultsFound: sentry.captureException() - raise DuplicateFileException( - file_name=name, - request_id=request_id - ) + raise DuplicateFileException(file_name=name, request_id=request_id) - super(Files, self).__init__(request_id, - privacy, - date_modified, - is_editable) + super(Files, self).__init__(request_id, privacy, date_modified, is_editable) self.name = name self.mime_type = mime_type self.title = title @@ -1487,23 +1614,17 @@ class Links(Responses): title - a string containing the title of a link url - a string containing the url link """ + __tablename__ = response_type.LINK - __mapper_args__ = {'polymorphic_identity': response_type.LINK} + __mapper_args__ = {"polymorphic_identity": response_type.LINK} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) title = db.Column(db.String) url = db.Column(db.String) - def __init__(self, - request_id, - privacy, - title, - url, - date_modified=None, - is_editable=False): - super(Links, self).__init__(request_id, - privacy, - date_modified, - is_editable) + def __init__( + self, request_id, privacy, title, url, date_modified=None, is_editable=False + ): + super(Links, self).__init__(request_id, privacy, date_modified, is_editable) self.title = title self.url = url @@ -1519,21 +1640,18 @@ class Instructions(Responses): id - an integer that is the primary key of Instructions content - a string containing the content of an instruction """ + __tablename__ = response_type.INSTRUCTIONS - __mapper_args__ = {'polymorphic_identity': response_type.INSTRUCTIONS} + __mapper_args__ = {"polymorphic_identity": response_type.INSTRUCTIONS} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) content = db.Column(db.String) - def __init__(self, - request_id, - privacy, - content, - date_modified=None, - is_editable=False): - super(Instructions, self).__init__(request_id, - privacy, - date_modified, - is_editable) + def __init__( + self, request_id, privacy, content, date_modified=None, is_editable=False + ): + super(Instructions, self).__init__( + request_id, privacy, date_modified, is_editable + ) self.content = content @property @@ -1552,8 +1670,9 @@ class Emails(Responses): subject - a string containing the subject of an email email_content - a string containing the content of an email """ + __tablename__ = response_type.EMAIL - __mapper_args__ = {'polymorphic_identity': response_type.EMAIL} + __mapper_args__ = {"polymorphic_identity": response_type.EMAIL} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) to = db.Column(db.String) cc = db.Column(db.String) @@ -1561,20 +1680,19 @@ class Emails(Responses): subject = db.Column(db.String(5000)) body = db.Column(db.String) - def __init__(self, - request_id, - privacy, - to, - cc, - bcc, - subject, - body, - date_modified=None, - is_editable=False): - super(Emails, self).__init__(request_id, - privacy, - date_modified, - is_editable) + def __init__( + self, + request_id, + privacy, + to, + cc, + bcc, + subject, + body, + date_modified=None, + is_editable=False, + ): + super(Emails, self).__init__(request_id, privacy, date_modified, is_editable) self.to = to self.cc = cc self.bcc = bcc @@ -1587,10 +1705,7 @@ def preview(self): @property def val_for_events(self): - return { - 'privacy': self.privacy, - 'body': self.body, - } + return {"privacy": self.privacy, "body": self.body} class Envelopes(Responses): @@ -1600,23 +1715,16 @@ class Envelopes(Responses): id - an integer that is the primary key of Envelopes (FK to Responses) latex - the latex used to generate the envelope PDF """ + __tablename__ = response_type.ENVELOPE - __mapper_args__ = {'polymorphic_identity': response_type.ENVELOPE} + __mapper_args__ = {"polymorphic_identity": response_type.ENVELOPE} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) latex = db.Column(db.String) - def __init__(self, - request_id, - privacy, - latex, - date_modified=None, - is_editable=False): - super(Envelopes, self).__init__( - request_id, - privacy, - date_modified, - is_editable - ) + def __init__( + self, request_id, privacy, latex, date_modified=None, is_editable=False + ): + super(Envelopes, self).__init__(request_id, privacy, date_modified, is_editable) self.latex = latex @property @@ -1634,26 +1742,32 @@ class EnvelopeTemplates(db.Model): title - a short descriptor for the envelope template template_name - the name of the template to be loaded from the filesystem """ - __tablename__ = 'envelope_templates' + + __tablename__ = "envelope_templates" id = db.Column(db.Integer, primary_key=True) - agency_ein = db.Column(db.String(4), db.ForeignKey('agencies.ein')) + agency_ein = db.Column(db.String(4), db.ForeignKey("agencies.ein")) title = db.Column(db.String, nullable=False) template_name = db.Column(db.String, nullable=False) @classmethod def populate(cls, csv_name=None): - filename = csv_name or current_app.config['ENVELOPE_TEMPLATES_DATA'] + filename = csv_name or current_app.config["ENVELOPE_TEMPLATES_DATA"] print(filename) - with open(filename, 'r') as data: + with open(filename, "r") as data: dictreader = csv.DictReader(data) for row in dictreader: - if EnvelopeTemplates.query.filter_by(agency_ein=row['agency_ein'], - title=row['title'], - template_name=row['template_name']).one_or_none() is None: + if ( + EnvelopeTemplates.query.filter_by( + agency_ein=row["agency_ein"], + title=row["title"], + template_name=row["template_name"], + ).one_or_none() + is None + ): template = EnvelopeTemplates( - agency_ein=row['agency_ein'], - title=row['title'], - template_name=row['template_name'] + agency_ein=row["agency_ein"], + title=row["title"], + template_name=row["template_name"], ) db.session.add(template) @@ -1667,23 +1781,17 @@ class Letters(Responses): id - an integer that is the primary key of Letters (FK to Responses) content - A string containing the content of a letter (HTML Formatted) """ + __tablename__ = response_type.LETTER - __mapper_args__ = {'polymorphic_identity': response_type.LETTER} + __mapper_args__ = {"polymorphic_identity": response_type.LETTER} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) title = db.Column(db.String, nullable=False) content = db.Column(db.String) - def __init__(self, - request_id, - privacy, - title, - content, - date_modified=None, - is_editable=False): - super(Letters, self).__init__(request_id, - privacy, - date_modified, - is_editable) + def __init__( + self, request_id, privacy, title, content, date_modified=None, is_editable=False + ): + super(Letters, self).__init__(request_id, privacy, date_modified, is_editable) self.title = title self.content = content @@ -1705,36 +1813,46 @@ class LetterTemplates(db.Model): Reason are based off the Law Department's responses. """ - __tablename__ = 'letter_templates' + + __tablename__ = "letter_templates" id = db.Column(db.Integer, primary_key=True) - type_ = db.Column(db.Enum( - determination_type.ACKNOWLEDGMENT, - determination_type.EXTENSION, - determination_type.CLOSING, - determination_type.DENIAL, - determination_type.REOPENING, - response_type.LETTER, - name="letter_type" - ), nullable=False, name='type') - agency_ein = db.Column(db.String(4), db.ForeignKey('agencies.ein')) + type_ = db.Column( + db.Enum( + determination_type.ACKNOWLEDGMENT, + determination_type.EXTENSION, + determination_type.CLOSING, + determination_type.DENIAL, + determination_type.REOPENING, + response_type.LETTER, + name="letter_type", + ), + nullable=False, + name="type", + ) + agency_ein = db.Column(db.String(4), db.ForeignKey("agencies.ein")) title = db.Column(db.String, nullable=False) content = db.Column(db.String, nullable=False) @classmethod def populate(cls, csv_name=None): - filename = csv_name or current_app.config['LETTER_TEMPLATES_DATA'] - with open(filename, 'r') as data: + filename = csv_name or current_app.config["LETTER_TEMPLATES_DATA"] + with open(filename, "r") as data: dictreader = csv.DictReader(data) for row in dictreader: - if LetterTemplates.query.filter_by(type_=row['type'], - agency_ein=row['agency_ein'], - title=row['title'], - content=row['content']).one_or_none() is None: + if ( + LetterTemplates.query.filter_by( + type_=row["type"], + agency_ein=row["agency_ein"], + title=row["title"], + content=row["content"], + ).one_or_none() + is None + ): template = LetterTemplates( - type_=row['type'], - agency_ein=row['agency_ein'], - title=row['title'], - content=row['content'] + type_=row["type"], + agency_ein=row["agency_ein"], + title=row["title"], + content=row["content"], ) db.session.add(template) @@ -1759,42 +1877,56 @@ class Determinations(Responses): reopening | new estimated date of completion | N/A """ + __tablename__ = response_type.DETERMINATION - __mapper_args__ = {'polymorphic_identity': response_type.DETERMINATION} + __mapper_args__ = {"polymorphic_identity": response_type.DETERMINATION} id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) - dtype = db.Column(db.Enum( - determination_type.DENIAL, - determination_type.ACKNOWLEDGMENT, - determination_type.EXTENSION, - determination_type.CLOSING, - determination_type.REOPENING, - name="determination_type" - ), nullable=False) + dtype = db.Column( + db.Enum( + determination_type.DENIAL, + determination_type.ACKNOWLEDGMENT, + determination_type.EXTENSION, + determination_type.CLOSING, + determination_type.REOPENING, + name="determination_type", + ), + nullable=False, + ) reason = db.Column(db.String) # nullable only for acknowledge and re-opening date = db.Column(db.DateTime) # nullable only for denial, closing - def __init__(self, - request_id, - privacy, # TODO: always RELEASE_AND_PUBLIC? - dtype, - reason, - date=None, - date_modified=None, - is_editable=False): - super(Determinations, self).__init__(request_id, - privacy, - date_modified, - is_editable) + def __init__( + self, + request_id, + privacy, # TODO: always RELEASE_AND_PUBLIC? + dtype, + reason, + date=None, + date_modified=None, + is_editable=False, + ): + super(Determinations, self).__init__( + request_id, privacy, date_modified, is_editable + ) self.dtype = dtype - if dtype not in (determination_type.ACKNOWLEDGMENT, - determination_type.REOPENING) and reason is None: - raise InvalidDeterminationException(request_id=request_id, dtype=dtype, missing_field='reason') + if ( + dtype + not in (determination_type.ACKNOWLEDGMENT, determination_type.REOPENING) + and reason is None + ): + raise InvalidDeterminationException( + request_id=request_id, dtype=dtype, missing_field="reason" + ) self.reason = reason - if dtype not in (determination_type.DENIAL, - determination_type.CLOSING) and date is None: - raise InvalidDeterminationException(request_id=request_id, dtype=dtype, missing_field='date') + if ( + dtype not in (determination_type.DENIAL, determination_type.CLOSING) + and date is None + ): + raise InvalidDeterminationException( + request_id=request_id, dtype=dtype, missing_field="date" + ) self.date = date @property @@ -1803,13 +1935,13 @@ def preview(self): @property def val_for_events(self): - val = { - 'reason': self.reason - } - if self.dtype in (determination_type.ACKNOWLEDGMENT, - determination_type.EXTENSION, - determination_type.REOPENING): - val['due_date'] = self.date.isoformat() + val = {"reason": self.reason} + if self.dtype in ( + determination_type.ACKNOWLEDGMENT, + determination_type.EXTENSION, + determination_type.REOPENING, + ): + val["due_date"] = self.date.isoformat() return val @@ -1825,19 +1957,18 @@ class CommunicationMethods(db.Model): method_id - an integer that is a primary key of CommunicationMethods (FK to Responses) method_type - enum ('letters', 'emails') method associated with the response """ - __tablename__ = 'communication_methods' + + __tablename__ = "communication_methods" response_id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) method_id = db.Column(db.Integer, db.ForeignKey(Responses.id), primary_key=True) - method_type = db.Column(db.Enum( - response_type.LETTER, - response_type.EMAIL, - name='communication_method_type' - ), nullable=True) - - def __init__(self, - response_id, - method_id, - method_type): + method_type = db.Column( + db.Enum( + response_type.LETTER, response_type.EMAIL, name="communication_method_type" + ), + nullable=True, + ) + + def __init__(self, response_id, method_id, method_type): self.response_id = response_id self.method_id = method_id self.method_type = method_type @@ -1856,9 +1987,10 @@ class CustomRequestForms(db.Model): category - an integer to separate different types of custom forms for an agency minimum_required - an integer to dictates the minimum amount of fields required for a successful submission """ - __tablename__ = 'custom_request_forms' + + __tablename__ = "custom_request_forms" id = db.Column(db.Integer, primary_key=True) - agency_ein = db.Column(db.String(4), db.ForeignKey('agencies.ein'), nullable=False) + agency_ein = db.Column(db.String(4), db.ForeignKey("agencies.ein"), nullable=False) form_name = db.Column(db.String, nullable=False) form_description = db.Column(db.String, nullable=False) field_definitions = db.Column(JSONB, nullable=False) @@ -1871,24 +2003,32 @@ def populate(cls, json_name=None): """ Automatically populate the custom_request_forms table for the OpenRecords application. """ - filename = json_name or current_app.config['CUSTOM_REQUEST_FORMS_DATA'] - with open(filename, 'r') as data: + filename = json_name or current_app.config["CUSTOM_REQUEST_FORMS_DATA"] + with open(filename, "r") as data: data = json.load(data) - for form in data['custom_request_forms']: - if CustomRequestForms.query.filter_by(agency_ein=form['agency_ein'], - form_name=form['form_name']).first() is not None: - warn("Duplicate custom_request_form ({}); Row not imported".format(form['agency_ein']), - category=UserWarning) + for form in data["custom_request_forms"]: + if ( + CustomRequestForms.query.filter_by( + agency_ein=form["agency_ein"], form_name=form["form_name"] + ).first() + is not None + ): + warn( + "Duplicate custom_request_form ({}); Row not imported".format( + form["agency_ein"] + ), + category=UserWarning, + ) continue custom_request_form = cls( - agency_ein=form['agency_ein'], - form_name=form['form_name'], - form_description=form['form_description'], - field_definitions=form['field_definitions'], - repeatable=form['repeatable'], - category=form['category'], - minimum_required=form['minimum_required'] + agency_ein=form["agency_ein"], + form_name=form["form_name"], + form_description=form["form_description"], + field_definitions=form["field_definitions"], + repeatable=form["repeatable"], + category=form["category"], + minimum_required=form["minimum_required"], ) db.session.add(custom_request_form) db.session.commit() diff --git a/app/request/views.py b/app/request/views.py index 77fd20f54..2c1ea372b 100644 --- a/app/request/views.py +++ b/app/request/views.py @@ -15,31 +15,18 @@ flash, Markup, jsonify, - abort + abort, ) from flask_login import current_user from sqlalchemy import any_ from sqlalchemy.orm.exc import NoResultFound from werkzeug.utils import escape -from app.constants import ( - request_status, - permission, - HIDDEN_AGENCIES -) -from app.lib.date_utils import ( - DEFAULT_YEARS_HOLIDAY_LIST, - get_holidays_date_list -) -from app.lib.permission_utils import ( - is_allowed -) +from app.constants import request_status, permission, HIDDEN_AGENCIES +from app.lib.date_utils import DEFAULT_YEARS_HOLIDAY_LIST, get_holidays_date_list +from app.lib.permission_utils import is_allowed from app.lib.utils import InvalidUserException, eval_request_bool -from app.models import ( - Requests, - Agencies, - UserRequests -) +from app.models import Requests, Agencies, UserRequests from app.request import request from app.request.forms import ( PublicUserRequestForm, @@ -56,19 +43,19 @@ SearchRequestsForm, CloseRequestForm, ContactAgencyForm, - ReopenRequestForm + ReopenRequestForm, ) from app.request.utils import ( create_request, handle_upload_no_id, get_address, send_confirmation_email, - create_contact_record + create_contact_record, ) from app.user_request.forms import ( AddUserRequestForm, EditUserRequestForm, - RemoveUserRequestForm + RemoveUserRequestForm, ) from app.user_request.utils import get_current_point_of_contact from app import sentry @@ -117,10 +104,18 @@ def new(): form.request_file.validate(form) upload_path = handle_upload_no_id(form.request_file) if form.request_file.errors: - return render_template(new_request_template, form=form, site_key=site_key) - - custom_metadata = json.loads(flask_request.form.get("custom-request-forms-data", {})) - tz_name = flask_request.form["tz-name"] if flask_request.form["tz-name"] else current_app.config["APP_TIMEZONE"] + return render_template( + new_request_template, form=form, site_key=site_key + ) + + custom_metadata = json.loads( + flask_request.form.get("custom-request-forms-data", {}) + ) + tz_name = ( + flask_request.form["tz-name"] + if flask_request.form["tz-name"] + else current_app.config["APP_TIMEZONE"] + ) if current_user.is_public: request_id = create_request( form.request_title.data, @@ -137,7 +132,9 @@ def new(): form.request_description.data, category=None, agency_ein=( - form.request_agency.data if form.request_agency.data != "None" else current_user.default_agency_ein + form.request_agency.data + if form.request_agency.data != "None" + else current_user.default_agency_ein ), submission=form.method_received.data, agency_date_submitted_local=form.request_date.data, @@ -175,14 +172,20 @@ def new(): current_request = Requests.query.filter_by(id=request_id).first() requester = current_request.requester - send_confirmation_email(request=current_request, agency=current_request.agency, user=requester) + send_confirmation_email( + request=current_request, agency=current_request.agency, user=requester + ) if current_request.agency.is_active: if requester.email: - flashed_message_html = render_template("request/confirmation_email.html") + flashed_message_html = render_template( + "request/confirmation_email.html" + ) flash(Markup(flashed_message_html), category="success") else: - flashed_message_html = render_template("request/confirmation_non_email.html") + flashed_message_html = render_template( + "request/confirmation_non_email.html" + ) flash(Markup(flashed_message_html), category="warning") return redirect(url_for("request.view", request_id=request_id)) @@ -191,29 +194,41 @@ def new(): "request/non_portal_agency_message.html", agency=current_request.agency ) flash(Markup(flashed_message_html), category="warning") - return redirect(url_for("request.non_portal_agency", agency_name=current_request.agency.name)) + return redirect( + url_for( + "request.non_portal_agency", agency_name=current_request.agency.name + ) + ) return render_template( - new_request_template, form=form, site_key=site_key, kiosk_mode=kiosk_mode, category=category, agency=agency, title=title + new_request_template, + form=form, + site_key=site_key, + kiosk_mode=kiosk_mode, + category=category, + agency=agency, + title=title, ) -@request.route('/view_all', methods=['GET']) +@request.route("/view_all", methods=["GET"]) def view_all(): - user_agencies = current_user.get_agencies if current_user.is_agency else '' + user_agencies = current_user.get_agencies if current_user.is_agency else "" return render_template( - 'request/all.html', + "request/all.html", form=SearchRequestsForm(), - holidays=sorted(get_holidays_date_list( - datetime.utcnow().year, - (datetime.utcnow() + rd(years=DEFAULT_YEARS_HOLIDAY_LIST)).year) + holidays=sorted( + get_holidays_date_list( + datetime.utcnow().year, + (datetime.utcnow() + rd(years=DEFAULT_YEARS_HOLIDAY_LIST)).year, + ) ), - user_agencies=user_agencies + user_agencies=user_agencies, ) -@request.route('/', methods=['GET']) -@request.route('/view/', methods=['GET']) +@request.route("/", methods=["GET"]) +@request.route("/view/", methods=["GET"]) def view(request_id): """ This function is for testing purposes of the view a request back until backend functionality is implemented. @@ -232,16 +247,20 @@ def view(request_id): sentry.captureException() return abort(404) - holidays = sorted(get_holidays_date_list( - datetime.utcnow().year, - (datetime.utcnow() + rd(years=DEFAULT_YEARS_HOLIDAY_LIST)).year) + holidays = sorted( + get_holidays_date_list( + datetime.utcnow().year, + (datetime.utcnow() + rd(years=DEFAULT_YEARS_HOLIDAY_LIST)).year, + ) ) active_users = [] assigned_users = [] if current_user.is_agency: for agency_user in current_request.agency.active_users: - if not agency_user in current_request.agency.administrators and (agency_user != current_user): + if not agency_user in current_request.agency.administrators and ( + agency_user != current_user + ): # populate list of assigned users that can be removed from a request if agency_user in current_request.agency_users: assigned_users.append(agency_user) @@ -250,100 +269,123 @@ def view(request_id): active_users.append(agency_user) permissions = { - 'acknowledge': permission.ACKNOWLEDGE, - 'deny': permission.DENY, - 'extend': permission.EXTEND, - 'close': permission.CLOSE, - 're_open': permission.RE_OPEN, - 'add_file': permission.ADD_FILE, - 'edit_file_privacy': permission.EDIT_FILE_PRIVACY, - 'delete_file': permission.DELETE_FILE, - 'add_note': permission.ADD_NOTE, - 'edit_note_privacy': permission.EDIT_NOTE_PRIVACY, - 'delete_note': permission.DELETE_NOTE, - 'add_link': permission.ADD_LINK, - 'edit_link_privacy': permission.EDIT_LINK_PRIVACY, - 'delete_link': permission.DELETE_LINK, - 'add_instructions': permission.ADD_OFFLINE_INSTRUCTIONS, - 'edit_instructions_privacy': permission.EDIT_OFFLINE_INSTRUCTIONS_PRIVACY, - 'delete_instructions': permission.DELETE_OFFLINE_INSTRUCTIONS, - 'generate_letter': permission.GENERATE_LETTER, - 'add_user': permission.ADD_USER_TO_REQUEST, - 'edit_user': permission.EDIT_USER_REQUEST_PERMISSIONS, - 'remove_user': permission.REMOVE_USER_FROM_REQUEST, - 'edit_title': permission.EDIT_TITLE, - 'edit_title_privacy': permission.CHANGE_PRIVACY_TITLE, - 'edit_agency_request_summary': permission.EDIT_AGENCY_REQUEST_SUMMARY, - 'edit_agency_request_summary_privacy': permission.CHANGE_PRIVACY_AGENCY_REQUEST_SUMMARY, - 'edit_requester_info': permission.EDIT_REQUESTER_INFO + "acknowledge": permission.ACKNOWLEDGE, + "deny": permission.DENY, + "extend": permission.EXTEND, + "close": permission.CLOSE, + "re_open": permission.RE_OPEN, + "add_file": permission.ADD_FILE, + "edit_file_privacy": permission.EDIT_FILE_PRIVACY, + "delete_file": permission.DELETE_FILE, + "add_note": permission.ADD_NOTE, + "edit_note_privacy": permission.EDIT_NOTE_PRIVACY, + "delete_note": permission.DELETE_NOTE, + "add_link": permission.ADD_LINK, + "edit_link_privacy": permission.EDIT_LINK_PRIVACY, + "delete_link": permission.DELETE_LINK, + "add_instructions": permission.ADD_OFFLINE_INSTRUCTIONS, + "edit_instructions_privacy": permission.EDIT_OFFLINE_INSTRUCTIONS_PRIVACY, + "delete_instructions": permission.DELETE_OFFLINE_INSTRUCTIONS, + "generate_letter": permission.GENERATE_LETTER, + "add_user": permission.ADD_USER_TO_REQUEST, + "edit_user": permission.EDIT_USER_REQUEST_PERMISSIONS, + "remove_user": permission.REMOVE_USER_FROM_REQUEST, + "edit_title": permission.EDIT_TITLE, + "edit_title_privacy": permission.CHANGE_PRIVACY_TITLE, + "edit_agency_request_summary": permission.EDIT_AGENCY_REQUEST_SUMMARY, + "edit_agency_request_summary_privacy": permission.CHANGE_PRIVACY_AGENCY_REQUEST_SUMMARY, + "edit_requester_info": permission.EDIT_REQUESTER_INFO, } # Build permissions dictionary for checking on the front-end. for key, val in permissions.items(): - if current_user.is_anonymous or not current_request.user_requests.filter_by( - user_guid=current_user.guid).first(): + if ( + current_user.is_anonymous + or not current_request.user_requests.filter_by( + user_guid=current_user.guid + ).first() + ): permissions[key] = False else: - permissions[key] = is_allowed(current_user, request_id, val) if not current_user.is_anonymous else False + permissions[key] = ( + is_allowed(current_user, request_id, val) + if not current_user.is_anonymous + else False + ) # Build dictionary of current permissions for all assigned users. assigned_user_permissions = {} for u in assigned_users: - assigned_user_permissions[u.guid] = UserRequests.query.filter_by( - request_id=request_id, user_guid=u.guid).one().get_permission_choice_indices() + assigned_user_permissions[u.guid] = ( + UserRequests.query.filter_by(request_id=request_id, user_guid=u.guid) + .one() + .get_permission_choice_indices() + ) point_of_contact = get_current_point_of_contact(request_id) if point_of_contact: - current_point_of_contact = {'user_guid': point_of_contact.user_guid} + current_point_of_contact = {"user_guid": point_of_contact.user_guid} else: - current_point_of_contact = {'user_guid': ''} + current_point_of_contact = {"user_guid": ""} # Determine if the Agency Request Summary should be shown. show_agency_request_summary = False - if current_user in current_request.agency_users \ - or current_request.agency_request_summary \ - and (current_request.requester == current_user - and current_request.status == request_status.CLOSED - and not current_request.privacy['agency_request_summary'] - or current_request.status == request_status.CLOSED - and current_request.agency_request_summary_release_date - and current_request.agency_request_summary_release_date - < datetime.utcnow() - and not current_request.privacy['agency_request_summary']): + if ( + current_user in current_request.agency_users + or current_request.agency_request_summary + and ( + current_request.requester == current_user + and current_request.status == request_status.CLOSED + and not current_request.privacy["agency_request_summary"] + or current_request.status == request_status.CLOSED + and current_request.agency_request_summary_release_date + and current_request.agency_request_summary_release_date < datetime.utcnow() + and not current_request.privacy["agency_request_summary"] + ) + ): show_agency_request_summary = True - # Determine if the title should be shown. - show_title = (current_user in current_request.agency_users or - current_request.requester == current_user or - not current_request.privacy['title']) - # Determine if "Generate Letter" functionality is enabled for the agency. - if 'letters' in current_request.agency.agency_features: - generate_letters_enabled = current_request.agency.agency_features['letters']['generate_letters'] + if "letters" in current_request.agency.agency_features: + generate_letters_enabled = current_request.agency.agency_features["letters"][ + "generate_letters" + ] else: generate_letters_enabled = False # Determine if custom request forms are enabled - if 'enabled' in current_request.agency.agency_features['custom_request_forms']: - custom_request_forms_enabled = current_request.agency.agency_features['custom_request_forms']['enabled'] + if "enabled" in current_request.agency.agency_features["custom_request_forms"]: + custom_request_forms_enabled = current_request.agency.agency_features[ + "custom_request_forms" + ]["enabled"] else: custom_request_forms_enabled = False # Determine if custom request form panels should be expanded by default - if 'expand_by_default' in current_request.agency.agency_features['custom_request_forms']: - expand_by_default = current_request.agency.agency_features['custom_request_forms']['expand_by_default'] + if ( + "expand_by_default" + in current_request.agency.agency_features["custom_request_forms"] + ): + expand_by_default = current_request.agency.agency_features[ + "custom_request_forms" + ]["expand_by_default"] else: expand_by_default = False # Determine if request description should be hidden when custom forms are enabled - if 'description_hidden_by_default' in current_request.agency.agency_features['custom_request_forms']: - description_hidden_by_default = current_request.agency.agency_features['custom_request_forms']['description_hidden_by_default'] + if ( + "description_hidden_by_default" + in current_request.agency.agency_features["custom_request_forms"] + ): + description_hidden_by_default = current_request.agency.agency_features[ + "custom_request_forms" + ]["description_hidden_by_default"] else: description_hidden_by_default = False return render_template( - 'request/view_request.html', + "request/view_request.html", request=current_request, status=request_status, agency_users=current_request.agency_users, @@ -355,12 +397,24 @@ def view(request_id): remove_user_request_form=RemoveUserRequestForm(assigned_users), add_user_request_form=AddUserRequestForm(active_users), edit_user_request_form=EditUserRequestForm(assigned_users), - generate_acknowledgment_letter_form=GenerateAcknowledgmentLetterForm(current_request.agency.ein), - generate_denial_letter_form=GenerateDenialLetterForm(current_request.agency.ein), - generate_closing_letter_form=GenerateClosingLetterForm(current_request.agency.ein), - generate_extension_letter_form=GenerateExtensionLetterForm(current_request.agency.ein), - generate_envelope_form=GenerateEnvelopeForm(current_request.agency_ein, current_request.requester), - generate_response_letter_form=GenerateResponseLetterForm(current_request.agency.ein), + generate_acknowledgment_letter_form=GenerateAcknowledgmentLetterForm( + current_request.agency.ein + ), + generate_denial_letter_form=GenerateDenialLetterForm( + current_request.agency.ein + ), + generate_closing_letter_form=GenerateClosingLetterForm( + current_request.agency.ein + ), + generate_extension_letter_form=GenerateExtensionLetterForm( + current_request.agency.ein + ), + generate_envelope_form=GenerateEnvelopeForm( + current_request.agency_ein, current_request.requester + ), + generate_response_letter_form=GenerateResponseLetterForm( + current_request.agency.ein + ), assigned_user_permissions=assigned_user_permissions, current_point_of_contact=current_point_of_contact, holidays=holidays, @@ -368,51 +422,61 @@ def view(request_id): active_users=active_users, permissions=permissions, show_agency_request_summary=show_agency_request_summary, - show_title=show_title, + show_title=current_request.show_title, is_requester=(current_request.requester == current_user), permissions_length=len(permission.ALL), generate_letters_enabled=generate_letters_enabled, - custom_request_forms_enabled = custom_request_forms_enabled, + custom_request_forms_enabled=custom_request_forms_enabled, expand_by_default=expand_by_default, - description_hidden_by_default=description_hidden_by_default + description_hidden_by_default=description_hidden_by_default, ) -@request.route('/non_portal_agency/', methods=['GET']) +@request.route("/non_portal_agency/", methods=["GET"]) def non_portal_agency(agency_name): """ This function handles messaging to the requester if they submitted a request to a non-portal agency. :return: redirect to non_portal_agency page. """ - return render_template('request/non_partner_request.html', agency_name=agency_name) + return render_template("request/non_partner_request.html", agency_name=agency_name) -@request.route('/agencies', methods=['GET']) +@request.route("/agencies", methods=["GET"]) def get_agencies_as_choices(): """ Get selected category value from the request body and generate a list of sorted agencies from the category. :return: list of agency choices """ - if flask_request.args['category']: + if flask_request.args["category"]: # TODO: is sorted faster than orderby? choices = sorted( - [(agencies.ein, agencies.name) - for agencies in Agencies.query.filter( - flask_request.args['category'] == any_(Agencies.categories) - ).all() if agencies.ein not in HIDDEN_AGENCIES], - key=lambda x: x[1]) + [ + (agencies.ein, agencies.name) + for agencies in Agencies.query.filter( + flask_request.args["category"] == any_(Agencies.categories) + ).all() + if agencies.ein not in HIDDEN_AGENCIES + ], + key=lambda x: x[1], + ) else: choices = sorted( - [(agencies.ein, agencies.name) - for agencies in Agencies.query.all() if agencies.ein not in HIDDEN_AGENCIES], - key=lambda x: x[1]) - choices.insert(0, ('', '')) # Insert blank option at the beginning of choices to prevent auto selection + [ + (agencies.ein, agencies.name) + for agencies in Agencies.query.all() + if agencies.ein not in HIDDEN_AGENCIES + ], + key=lambda x: x[1], + ) + choices.insert( + 0, ("", "") + ) # Insert blank option at the beginning of choices to prevent auto selection return jsonify(choices) -@request.route('/contact/', methods=['POST']) +@request.route("/contact/", methods=["POST"]) def contact_agency(request_id): """ This function handles contacting the agency about a request as a requester. @@ -422,13 +486,18 @@ def contact_agency(request_id): form = ContactAgencyForm(current_request) del form.subject if form.validate_on_submit(): - create_contact_record(current_request, - flask_request.form['first_name'], - flask_request.form['last_name'], - flask_request.form['email'], - "Inquiry about {}".format(request_id), - flask_request.form['message']) - flash('Your message has been sent.', category='success') + create_contact_record( + current_request, + flask_request.form["first_name"], + flask_request.form["last_name"], + flask_request.form["email"], + "Inquiry about {}".format(request_id), + flask_request.form["message"], + ) + flash("Your message has been sent.", category="success") else: - flash('There was a problem sending your message. Please try again.', category='danger') - return redirect(url_for('request.view', request_id=request_id)) + flash( + "There was a problem sending your message. Please try again.", + category="danger", + ) + return redirect(url_for("request.view", request_id=request_id)) diff --git a/app/search/utils.py b/app/search/utils.py index 633ce4ecf..67fc63231 100644 --- a/app/search/utils.py +++ b/app/search/utils.py @@ -6,10 +6,7 @@ from sqlalchemy.orm import joinedload from app import es -from app.constants import ( - ES_DATETIME_FORMAT, - request_status -) +from app.constants import ES_DATETIME_FORMAT, request_status from app.lib.date_utils import utc_to_local, local_to_utc from app.lib.utils import InvalidUserException from app.models import Requests, Agencies @@ -17,7 +14,7 @@ MAX_RESULT_SIZE, ES_DATE_RANGE_FORMAT, DT_DATE_RANGE_FORMAT, - MOCK_EMPTY_ELASTICSEARCH_RESULT + MOCK_EMPTY_ELASTICSEARCH_RESULT, ) @@ -41,19 +38,14 @@ def delete_index(): """ Delete all elasticsearch indices, ignoring errors. """ - es.indices.delete( - current_app.config["ELASTICSEARCH_INDEX"], - ignore=[400, 404] - ) + es.indices.delete(current_app.config["ELASTICSEARCH_INDEX"], ignore=[400, 404]) def delete_docs(): """ Delete all elasticsearch request docs. """ - es.indices.refresh( - index=current_app.config["ELASTICSEARCH_INDEX"], - ) + es.indices.refresh(index=current_app.config["ELASTICSEARCH_INDEX"]) es.delete_by_query( index=current_app.config["ELASTICSEARCH_INDEX"], doc_type="request", @@ -79,40 +71,21 @@ def create_index(): "analyzer": "english", "fields": { # for sorting by title - "keyword": { - "type": "keyword", - } - } - }, - "description": { - "type": "text", - "analyzer": "english" + "keyword": {"type": "keyword"} + }, }, + "description": {"type": "text", "analyzer": "english"}, "agency_request_summary": { "type": "text", - "analyzer": "english" - }, - "requester_id": { - "type": "keyword", - }, - "title_private": { - "type": "boolean", - }, - "agency_request_summary_private": { - "type": "boolean", - }, - "agency_ein": { - "type": "keyword", - }, - "agency_name": { - "type": "keyword", - }, - "agency_acronym": { - "type": "keyword" - }, - "status": { - "type": "keyword", + "analyzer": "english", }, + "requester_id": {"type": "keyword"}, + "title_private": {"type": "boolean"}, + "agency_request_summary_private": {"type": "boolean"}, + "agency_ein": {"type": "keyword"}, + "agency_name": {"type": "keyword"}, + "agency_acronym": {"type": "keyword"}, + "status": {"type": "keyword"}, "date_submitted": { "type": "date", "format": "strict_date_hour_minute_second", @@ -133,13 +106,11 @@ def create_index(): "type": "date", "format": "strict_date_hour_minute_second", }, - "assigned_users": { - "type": "keyword" - } + "assigned_users": {"type": "keyword"}, } } } - } + }, ) @@ -151,82 +122,94 @@ def create_docs(): agency_eins = {a.ein: a for a in Agencies.query.filter_by(is_active=True).all()} #: :type: collections.Iterable[app.models.Requests] - requests = Requests.query.filter(Requests.agency_ein.in_(agency_eins.keys())).options( - joinedload(Requests.agency_users)).options(joinedload(Requests.requester)).all() + requests = ( + Requests.query.filter(Requests.agency_ein.in_(agency_eins.keys())) + .options(joinedload(Requests.agency_users)) + .options(joinedload(Requests.requester)) + .all() + ) operations = [] for r in requests: - date_received = r.date_created.strftime( - ES_DATETIME_FORMAT) if r.date_created < r.date_submitted else r.date_submitted.strftime( - ES_DATETIME_FORMAT) + date_received = ( + r.date_created.strftime(ES_DATETIME_FORMAT) + if r.date_created < r.date_submitted + else r.date_submitted.strftime(ES_DATETIME_FORMAT) + ) operation = { - '_op_type': 'create', - '_id': r.id, - 'title': r.title, - 'description': r.description, - 'agency_request_summary': r.agency_request_summary, - 'requester_name': r.requester.name, - 'requester_id': "{guid}".format(guid=r.requester.guid), - 'title_private': r.privacy['title'], - 'agency_request_summary_private': not r.agency_request_summary_released, - 'date_created': r.date_created.strftime(ES_DATETIME_FORMAT), - 'date_submitted': r.date_submitted.strftime(ES_DATETIME_FORMAT), - 'date_received': date_received, - 'date_due': r.due_date.strftime(ES_DATETIME_FORMAT), - 'submission': r.submission, - 'status': r.status, - 'agency_ein': r.agency_ein, - 'agency_acronym': agency_eins[r.agency_ein].acronym, - 'agency_name': agency_eins[r.agency_ein].name, - 'public_title': 'Private' if r.privacy['title'] else r.title, - 'assigned_users': ["{guid}".format(guid=user.guid) for user - in r.agency_users] + "_op_type": "create", + "_id": r.id, + "title": r.title, + "description": r.description, + "agency_request_summary": r.agency_request_summary, + "requester_name": r.requester.name, + "requester_id": "{guid}".format(guid=r.requester.guid), + "title_private": r.privacy["title"], + "agency_request_summary_private": not r.agency_request_summary_released, + "date_created": r.date_created.strftime(ES_DATETIME_FORMAT), + "date_submitted": r.date_submitted.strftime(ES_DATETIME_FORMAT), + "date_received": date_received, + "date_due": r.due_date.strftime(ES_DATETIME_FORMAT), + "submission": r.submission, + "status": r.status, + "agency_ein": r.agency_ein, + "agency_acronym": agency_eins[r.agency_ein].acronym, + "agency_name": agency_eins[r.agency_ein].name, + "public_title": "Private" if not r.show_title else r.title, + "assigned_users": [ + "{guid}".format(guid=user.guid) for user in r.agency_users + ] # public_agency_request_summary } if r.date_closed is not None: - operation['date_closed'] = r.date_closed.strftime(ES_DATETIME_FORMAT) + operation["date_closed"] = r.date_closed.strftime(ES_DATETIME_FORMAT) operations.append(operation) num_success, _ = bulk( es, operations, index=current_app.config["ELASTICSEARCH_INDEX"], - doc_type='request', - chunk_size=current_app.config['ELASTICSEARCH_CHUNK_SIZE'], - raise_on_error=True + doc_type="request", + chunk_size=current_app.config["ELASTICSEARCH_CHUNK_SIZE"], + raise_on_error=True, ) - current_app.logger.info("Successfully created {num_success} of {total_num} docs.".format(num_success=num_success, - total_num=len(requests))) - - -def search_requests(query, - foil_id, - title, - agency_request_summary, - description, - requester_name, - date_rec_from, - date_rec_to, - date_due_from, - date_due_to, - date_closed_from, - date_closed_to, - agency_ein, - agency_user_guid, - open_, - closed, - in_progress, - due_soon, - overdue, - start, - sort_date_received, - sort_date_due, - sort_title, - tz_name, - size=None, - by_phrase=False, - highlight=False, - for_csv=False): + current_app.logger.info( + "Successfully created {num_success} of {total_num} docs.".format( + num_success=num_success, total_num=len(requests) + ) + ) + + +def search_requests( + query, + foil_id, + title, + agency_request_summary, + description, + requester_name, + date_rec_from, + date_rec_to, + date_due_from, + date_due_to, + date_closed_from, + date_closed_to, + agency_ein, + agency_user_guid, + open_, + closed, + in_progress, + due_soon, + overdue, + start, + sort_date_received, + sort_date_due, + sort_title, + tz_name, + size=None, + by_phrase=False, + highlight=False, + for_csv=False, +): """ The arguments of this function match the request parameters of the '/search/requests' endpoints. @@ -274,24 +257,29 @@ def search_requests(query, query = query.strip() # return no results if there is nothing to query by - if query and not any((foil_id, title, agency_request_summary, - description, requester_name)): + if query and not any( + (foil_id, title, agency_request_summary, description, requester_name) + ): return MOCK_EMPTY_ELASTICSEARCH_RESULT # if searching by foil-id, strip "FOIL-" if foil_id: - query = query.lstrip("FOIL-").lstrip('foil-') + query = query.lstrip("FOIL-").lstrip("foil-") # set sort (list of "field:direction" pairs) sort = [ - ':'.join((field, direction)) for field, direction in { - 'date_received': sort_date_received, - 'date_due': sort_date_due, - 'title.keyword': sort_title}.items() if direction in ("desc", "asc")] + ":".join((field, direction)) + for field, direction in { + "date_received": sort_date_received, + "date_due": sort_date_due, + "title.keyword": sort_title, + }.items() + if direction in ("desc", "asc") + ] # if no sort options are selected use date_received desc by default if len(sort) == 0: - sort = ['date_received:desc'] + sort = ["date_received:desc"] # set statuses (list of request statuses) if current_user.is_agency: @@ -300,24 +288,26 @@ def search_requests(query, request_status.CLOSED: closed, request_status.IN_PROGRESS: in_progress, request_status.DUE_SOON: due_soon, - request_status.OVERDUE: overdue + request_status.OVERDUE: overdue, } statuses = [s for s, b in statuses.items() if b] else: statuses = [] if open_: # Any request that isn't closed is considered open - statuses.extend([ - request_status.OPEN, - request_status.IN_PROGRESS, - request_status.DUE_SOON, - request_status.OVERDUE - ]) + statuses.extend( + [ + request_status.OPEN, + request_status.IN_PROGRESS, + request_status.DUE_SOON, + request_status.OVERDUE, + ] + ) if closed: statuses.append(request_status.CLOSED) # set matching type (full-text or phrase matching) - match_type = 'match_phrase' if by_phrase else 'match' + match_type = "match_phrase" if by_phrase else "match" # set date ranges def datestr_local_to_utc(datestr): @@ -326,41 +316,60 @@ def datestr_local_to_utc(datestr): ).strftime(DT_DATE_RANGE_FORMAT) date_ranges = [] - if any((date_rec_from, date_rec_to, date_due_from, date_due_to, date_closed_from, date_closed_to)): + if any( + ( + date_rec_from, + date_rec_to, + date_due_from, + date_due_to, + date_closed_from, + date_closed_to, + ) + ): range_filters = {} if date_rec_from or date_rec_to: - range_filters['date_received'] = {'format': ES_DATE_RANGE_FORMAT} + range_filters["date_received"] = {"format": ES_DATE_RANGE_FORMAT} if date_due_from or date_due_to: - range_filters['date_due'] = {'format': ES_DATE_RANGE_FORMAT} + range_filters["date_due"] = {"format": ES_DATE_RANGE_FORMAT} if date_closed_from or date_closed_to: - range_filters['date_closed'] = {'format': ES_DATE_RANGE_FORMAT} + range_filters["date_closed"] = {"format": ES_DATE_RANGE_FORMAT} if date_rec_from: - range_filters['date_received']['gte'] = datestr_local_to_utc(date_rec_from) + range_filters["date_received"]["gte"] = datestr_local_to_utc(date_rec_from) if date_rec_to: - range_filters['date_received']['lt'] = datestr_local_to_utc(date_rec_to) + range_filters["date_received"]["lt"] = datestr_local_to_utc(date_rec_to) if date_due_from: - range_filters['date_due']['gte'] = datestr_local_to_utc(date_due_from) + range_filters["date_due"]["gte"] = datestr_local_to_utc(date_due_from) if date_due_to: - range_filters['date_due']['lt'] = datestr_local_to_utc(date_due_to) + range_filters["date_due"]["lt"] = datestr_local_to_utc(date_due_to) if date_closed_from: - range_filters['date_closed']['gte'] = datestr_local_to_utc(date_closed_from) + range_filters["date_closed"]["gte"] = datestr_local_to_utc(date_closed_from) if date_closed_to: - range_filters['date_closed']['lte'] = datestr_local_to_utc(date_closed_to) + range_filters["date_closed"]["lte"] = datestr_local_to_utc(date_closed_to) if date_rec_from or date_rec_to: - date_ranges.append({'range': {'date_received': range_filters['date_received']}}) + date_ranges.append( + {"range": {"date_received": range_filters["date_received"]}} + ) if date_due_from or date_due_to: - date_ranges.append({'range': {'date_due': range_filters['date_due']}}) + date_ranges.append({"range": {"date_due": range_filters["date_due"]}}) if date_closed_from or date_closed_to: - date_ranges.append({'range': {'date_closed': range_filters['date_closed']}}) + date_ranges.append({"range": {"date_closed": range_filters["date_closed"]}}) # generate query dsl body query_fields = { - 'title': title, - 'description': description, - 'agency_request_summary': agency_request_summary, - 'requester_name': requester_name + "title": title, + "description": description, + "agency_request_summary": agency_request_summary, + "requester_name": requester_name, } - dsl_gen = RequestsDSLGenerator(query, query_fields, statuses, date_ranges, agency_ein, agency_user_guid, match_type) + dsl_gen = RequestsDSLGenerator( + query, + query_fields, + statuses, + date_ranges, + agency_ein, + agency_user_guid, + match_type, + ) if foil_id: dsl = dsl_gen.foil_id() else: @@ -384,10 +393,10 @@ def datestr_local_to_utc(datestr): highlight_fields[name] = {} dsl.update( { - 'highlight': { - 'pre_tags': [''], - 'post_tags': [''], - 'fields': highlight_fields + "highlight": { + "pre_tags": [''], + "post_tags": [""], + "fields": highlight_fields, } } ) @@ -399,26 +408,28 @@ def datestr_local_to_utc(datestr): if not for_csv: results = es.search( index=current_app.config["ELASTICSEARCH_INDEX"], - doc_type='request', + doc_type="request", body=dsl, - _source=['requester_id', - 'date_submitted', - 'date_due', - 'date_received', - 'date_created', - 'date_closed', - 'status', - 'agency_ein', - 'agency_name', - 'agency_acronym', - 'requester_name', - 'title_private', - 'agency_request_summary_private', - 'public_title', - 'title', - 'agency_request_summary', - 'description', - 'assigned_users'], + _source=[ + "requester_id", + "date_submitted", + "date_due", + "date_received", + "date_created", + "date_closed", + "status", + "agency_ein", + "agency_name", + "agency_acronym", + "requester_name", + "title_private", + "agency_request_summary_private", + "public_title", + "title", + "agency_request_summary", + "description", + "assigned_users", + ], size=result_set_size, from_=start, sort=sort, @@ -427,42 +438,44 @@ def datestr_local_to_utc(datestr): else: results = es.search( index=current_app.config["ELASTICSEARCH_INDEX"], - doc_type='request', - scroll='1m', + doc_type="request", + scroll="1m", body=dsl, - _source=['requester_id', - 'date_submitted', - 'date_due', - 'date_received', - 'date_created', - 'date_closed', - 'status', - 'agency_ein', - 'agency_name', - 'agency_acronym', - 'requester_name', - 'title_private', - 'agency_request_summary_private', - 'public_title', - 'title', - 'agency_request_summary', - 'description', - 'assigned_users'], + _source=[ + "requester_id", + "date_submitted", + "date_due", + "date_received", + "date_created", + "date_closed", + "status", + "agency_ein", + "agency_name", + "agency_acronym", + "requester_name", + "title_private", + "agency_request_summary_private", + "public_title", + "title", + "agency_request_summary", + "description", + "assigned_users", + ], size=result_set_size, from_=start, sort=sort, ) - sid = results['_scroll_id'] - scroll_size = results['hits']['total'] + sid = results["_scroll_id"] + scroll_size = results["hits"]["total"] - scroll_results = results['hits']['hits'] + scroll_results = results["hits"]["hits"] while scroll_size > 0: - results = es.scroll(scroll='1m', body={"scroll": "1m", "scroll_id": sid}) + results = es.scroll(scroll="1m", body={"scroll": "1m", "scroll_id": sid}) - scroll_size = len(results['hits']['hits']) + scroll_size = len(results["hits"]["hits"]) - scroll_results += results['hits']['hits'] + scroll_results += results["hits"]["hits"] return scroll_results # process highlights @@ -475,117 +488,108 @@ def datestr_local_to_utc(datestr): class RequestsDSLGenerator(object): """ Class for generating dicts representing query dsl bodies for searching request docs. """ - def __init__(self, query, query_fields, statuses, date_ranges, agency_ein, agency_user_guid, match_type): + def __init__( + self, + query, + query_fields, + statuses, + date_ranges, + agency_ein, + agency_user_guid, + match_type, + ): self.__query = query self.__query_fields = query_fields self.__statuses = statuses self.__agency_ein = agency_ein self.__match_type = match_type - self.__default_filters = [{'terms': {'status': statuses}}] + self.__default_filters = [{"terms": {"status": statuses}}] if date_ranges: self.__default_filters += date_ranges if agency_ein: - self.__default_filters.append({ - 'term': {'agency_ein': agency_ein} - }) + self.__default_filters.append({"term": {"agency_ein": agency_ein}}) if agency_user_guid: - self.__default_filters.append({ - 'term': {'assigned_users': agency_user_guid} - }) + self.__default_filters.append( + {"term": {"assigned_users": agency_user_guid}} + ) self.__filters = [] self.__conditions = [] self.requester_id = None def foil_id(self): - self.__filters = [{ - 'wildcard': { - '_uid': 'request#FOIL-*{}*'.format(self.__query) - } - }] + self.__filters = [ + {"wildcard": {"_uid": "request#FOIL-*{}*".format(self.__query)}} + ] return self.__must_query def agency_user(self): for name, use in self.__query_fields.items(): if use: - self.__filters = [ - {self.__match_type: {name: self.__query}} - ] + self.__filters = [{self.__match_type: {name: self.__query}}] self.__conditions.append(self.__must) return self.__should def anonymous_user(self): - if self.__query_fields['title']: + if self.__query_fields["title"]: self.__filters = [ - {self.__match_type: {'title': self.__query}}, - {'term': {'title_private': False}} + {self.__match_type: {"title": self.__query}}, + {"term": {"title_private": False}}, ] self.__conditions.append(self.__must) - if self.__query_fields['agency_request_summary']: + if self.__query_fields["agency_request_summary"]: self.__filters = [ - {self.__match_type: {'agency_request_summary': self.__query}}, - {'term': {'agency_request_summary_private': False}} + {self.__match_type: {"agency_request_summary": self.__query}}, + {"term": {"agency_request_summary_private": False}}, ] self.__conditions.append(self.__must) return self.__should def public_user(self): self.requester_id = current_user.get_id() - if self.__query_fields['title']: + if self.__query_fields["title"]: self.__filters = [ - {self.__match_type: {'title': self.__query}}, - {'bool': { - 'should': [ - {'term': {'requester_id': self.requester_id}}, - {'term': {'title_private': False}} - ] - }} + {self.__match_type: {"title": self.__query}}, + { + "bool": { + "should": [ + {"term": {"requester_id": self.requester_id}}, + {"term": {"title_private": False}}, + ] + } + }, ] self.__conditions.append(self.__must) - if self.__query_fields['agency_request_summary']: + if self.__query_fields["agency_request_summary"]: self.__filters = [ - {self.__match_type: {'agency_request_summary': self.__query}}, - {'term': {'agency_request_summary_private': False}} + {self.__match_type: {"agency_request_summary": self.__query}}, + {"term": {"agency_request_summary_private": False}}, ] self.__conditions.append(self.__must) - if self.__query_fields['description']: + if self.__query_fields["description"]: self.__filters = [ - {self.__match_type: {'description': self.__query}}, - {'term': {'requester_id': self.requester_id}}, + {self.__match_type: {"description": self.__query}}, + {"term": {"requester_id": self.requester_id}}, ] self.__conditions.append(self.__must) return self.__should def queryless(self): - self.__filters = [ - {'match_all': {}}, - ] + self.__filters = [{"match_all": {}}] return self.__must_query @property def __must_query(self): - return { - 'query': self.__must - } + return {"query": self.__must} @property def __must(self): - return { - 'bool': { - 'must': self.__get_filters() - } - } + return {"bool": {"must": self.__get_filters()}} @property def __should(self): - return { - 'query': { - 'bool': { - 'should': self.__conditions - } - } - } + return {"query": {"bool": {"should": self.__conditions}}} def __get_filters(self): return self.__filters + self.__default_filters @@ -610,7 +614,9 @@ def convert_dates(results, dt_format=None, tz_name=None): continue if tz_name: dt = utc_to_local(dt, tz_name) - hit["_source"][field] = dt.strftime(dt_format) if dt_format is not None else dt + hit["_source"][field] = ( + dt.strftime(dt_format) if dt_format is not None else dt + ) def _process_highlights(results, requester_id=None): @@ -625,17 +631,22 @@ def _process_highlights(results, requester_id=None): :param requester_id: id of requester as it is exists in results """ if not current_user.is_agency: - for hit in results['hits']['hits']: - is_requester = (requester_id == hit['_source']['requester_id'] - if requester_id - else False) - if ('title' in hit['highlight'] - and hit['_source']['title_private'] - and (current_user.is_anonymous or not is_requester)): - hit['highlight'].pop('title') - if ('agency_request_summary' in hit['highlight'] - and hit['_source']['agency_request_summary_private']): - hit['highlight'].pop('agency_request_summary') - if ('description' in hit['highlight'] - and not is_requester): - hit['highlight'].pop('description') + for hit in results["hits"]["hits"]: + is_requester = ( + requester_id == hit["_source"]["requester_id"] + if requester_id + else False + ) + if ( + "title" in hit["highlight"] + and hit["_source"]["title_private"] + and (current_user.is_anonymous or not is_requester) + ): + hit["highlight"].pop("title") + if ( + "agency_request_summary" in hit["highlight"] + and hit["_source"]["agency_request_summary_private"] + ): + hit["highlight"].pop("agency_request_summary") + if "description" in hit["highlight"] and not is_requester: + hit["highlight"].pop("description") From fac397bd7101666cb5ce2e2363366415d9d557d6 Mon Sep 17 00:00:00 2001 From: Joel Castillo Date: Wed, 14 Aug 2019 15:06:21 -0400 Subject: [PATCH 03/24] Prompt user to check for PII when submitting. Prompts anonymous and logged in public users to check for PII before submitting the FOIL request form. It does not currently affect agency users when submitting the form. Waiting for language for modal dialog. --- .../js/request/new-request-anon.js} | 18 ++++++---------- app/static/js/request/new-request-user.js | 21 +++++++++---------- app/templates/request/new_request_anon.html | 3 ++- 3 files changed, 18 insertions(+), 24 deletions(-) rename app/{templates/request/new_request_anon.js.html => static/js/request/new-request-anon.js} (97%) diff --git a/app/templates/request/new_request_anon.js.html b/app/static/js/request/new-request-anon.js similarity index 97% rename from app/templates/request/new_request_anon.js.html rename to app/static/js/request/new-request-anon.js index b22b09891..8594b7ed8 100644 --- a/app/templates/request/new_request_anon.js.html +++ b/app/static/js/request/new-request-anon.js @@ -1,5 +1,4 @@ - \ No newline at end of file + }); \ No newline at end of file diff --git a/app/static/js/request/new-request-user.js b/app/static/js/request/new-request-user.js index 9d85ad349..45251e252 100755 --- a/app/static/js/request/new-request-user.js +++ b/app/static/js/request/new-request-user.js @@ -77,12 +77,10 @@ $(document).ready(function () { $("#cancel-change-category-button").off().click(function () { $(targetId).val(previousValues[target - 1]); }); - } - else{ // otherwise render the form normally + } else { // otherwise render the form normally renderCustomRequestForm(target); } - } - else { + } else { renderCustomRequestForm(target); categorySelected = true; } @@ -176,8 +174,7 @@ $(document).ready(function () { // TODO: this or combine (see the other new-request-* js files) if ($("#request-file").parsley().isValid() === false) { $(".file-error").show(); - } - else { + } else { $(".file-error").hide(); } @@ -233,14 +230,16 @@ $(document).ready(function () { return; } - var ssnInTitle = checkSSN($('#request-title').val()); - var ssnInDescription = checkSSN($('#request-description').val()); - var ssnInCustomRequestForms = checkSSN($('#custom-request-forms-data').val()); - if ((ssnInTitle || ssnInDescription || ssnInCustomRequestForms) && showSsnWarning) { + e.preventDefault(); + $('#submit').show(); + $('#pii-warning-modal').modal('show'); + return; + + if (showPIIWarning) { e.preventDefault(); $('#submit').show(); $('#pii-warning-modal').modal('show'); - showSsnWarning = false; + showPIIWarning = false; return; } } diff --git a/app/templates/request/new_request_anon.html b/app/templates/request/new_request_anon.html index 99c44fbb5..8dcdb33ff 100644 --- a/app/templates/request/new_request_anon.html +++ b/app/templates/request/new_request_anon.html @@ -274,7 +274,8 @@

Address

src="{{ url_for('static', filename='js/plugins/jquery.timepicker.min.js') }}"> - {% include 'request/new_request_anon.js.html' %} + {# #} From 280a3c052d02c05826a039908ae99c11a90026c4 Mon Sep 17 00:00:00 2001 From: Joel Castillo Date: Wed, 14 Aug 2019 15:08:37 -0400 Subject: [PATCH 04/24] Remove recaptcha requirement --- app/templates/request/new_request_anon.html | 3 --- 1 file changed, 3 deletions(-) diff --git a/app/templates/request/new_request_anon.html b/app/templates/request/new_request_anon.html index 8dcdb33ff..5bd7299ed 100644 --- a/app/templates/request/new_request_anon.html +++ b/app/templates/request/new_request_anon.html @@ -276,9 +276,6 @@

Address

src="{{ url_for('static', filename='js/validation/custom_validators.js') }}"> - - {# #} {% endblock %} From 0e8da93c522ff4ef3ad747211a57c947fe4074fb Mon Sep 17 00:00:00 2001 From: Joel Castillo Date: Wed, 14 Aug 2019 17:01:24 -0400 Subject: [PATCH 05/24] Moving PII Warning modal to separate file --- app/templates/request/_pii_warning_modal.html | 24 +++++++++++++++++++ app/templates/request/new_request_user.html | 21 +--------------- 2 files changed, 25 insertions(+), 20 deletions(-) create mode 100644 app/templates/request/_pii_warning_modal.html diff --git a/app/templates/request/_pii_warning_modal.html b/app/templates/request/_pii_warning_modal.html new file mode 100644 index 000000000..da0fd1cd3 --- /dev/null +++ b/app/templates/request/_pii_warning_modal.html @@ -0,0 +1,24 @@ + \ No newline at end of file diff --git a/app/templates/request/new_request_user.html b/app/templates/request/new_request_user.html index e518848f2..bae40bac6 100644 --- a/app/templates/request/new_request_user.html +++ b/app/templates/request/new_request_user.html @@ -162,26 +162,7 @@

Request a Record



{{ form.submit(id="submit", class="btn-primary") }} - + {% include "request/_pii_warning_modal.html" %}