There was no explicit `order by` in the queries that list the products
of quotes and invoices, so PostgreSQL was free to use any order it
wanted. In this case, since was am grouping first by name, the result
was sorted by product name.
This is not an issue in most cases, albeit a bit rude to the user,
except for when the products *have* to in the same order the user
entered them, because they are monthly fees or something like that, that
must be ordered by month _number_, not by their _name_; the user will
usually input them in the correct order they want them on the invoice or
quote.
Sorting by *_product_id does *not* guarantee that they will always be
in insertion order, because the sequence can “wrap”, but i think i am
going to have bigger problems at that point.
When i wrote the functions to import contact, i already created a couple
of “temporary” functions to validate whether the input given from the
Excel files was correct according to the various domains used in the
relations, so i can know whether i can import that data.
I realized that i could do exactly the same when validating forms: check
that the value conforms to the domain, in the exact same way, so i can
make sure that the value will be accepted without duplicating the logic,
at the expense of a call to the database.
In an ideal world, i would use pg_input_is_valid, but this function is
only available in PostgreSQL 16 and Debian 12 uses PostgreSQL 15.
These functions are in the public schema because initially i wanted to
use them to also validate email, which is needed in the login form, but
then i recanted and kept the same email validation in Go, because
something felt off about using the database for that particular form,
but i do not know why.
This allows to import an Excel file exported from Holded, because it is
our own user case. When we have more customers, we will give out an
Excel template file to fill out.
Why XLSX files instead of CSV, for instance? First, because this is the
output from Holded, but even then we would have more trouble with CSV
than with XLSX because of Microsoft: they royally fucked up
interoperability when decided that CSV files, the files that only other
applications or programmers see, should be “localized”, and use a comma
or a **semicolon** to separate a **comma** separated file depending on
the locale’s decimal separator.
This is ridiculous because it means that CSV files created with an Excel
in USA uses comma while the same Excel but with a French locale expects
the fields to be separated by semicolon. And for no good reason,
either.
Since they fucked up so bad, decided to add a non-standard “meta” field
to specify the separator, writing a `sep=,` in the first line, but this
only works for reading, because saving the same file changes the
separator back to the locale-dependent character and removes the “meta”
field.
And since everyone expects to open spreadsheet with Excel, i can not
use CSV if i do not want a bunch of support tickets telling me that the
template is all in a single line.
I use an extremely old version of a xlsx reading library for golang[0]
because it is already available in Debian repositories, and the only
thing i want from it is to convert the convoluted XML file into a
string array.
Go is only responsible to read the file and dump its contents into a
temporary table, so that it can execute the PL/pgSQL function that will
actually move that data to the correct relations, much like add_contact
does but in batch.
In PostgreSQL version 16 they added a pg_input_is_valid function that
i would use to test whether input values really conform to domains,
but i will have to wait for Debian to pick up the new version.
Meanwhile, i use a couple of temporary functions, in lieu of nested
functions support in PostgreSQL.
Part of #45
[0]: https://github.com/tealeg/xlsx
These two fields are just for information purposes, as Numerus does not
have any way to wire transfer using these, but people might want to keep
these in the contact’s info as a convenience.
Since not every contact should have an IBAN, e.g., customers, and inside
SEPA (European Union and some more countries) the BIC is not required,
they are in two different relations in order to be optional without
using NULL.
For the IBAN i found an already made PostgreSQL module, but for BIC i
had to write a regular expression based on the information i gathered
from Wikipedia, because the ISO standard is not free.
These two parameters for the add_contact and edit_contact functions are
TEXT because i realized that these functions are intended to be used
from the web application, that only deals with texts, so the
ValueOrNil() function was unnecessarily complex and PostreSQL’s
functions were better suited to “convert” from TEXT to IBAN or BIC.
The same is true for EMAIL and URI domains, so i changed their parameter
types to TEXT too.
Closes#54.
We need to have contacts with just a name: we need to assign
freelancer’s quote as expense linked the government, but of course we
do not have a phone or email for that “contact”, much less a VATIN or
other tax details.
It is also interesting for other expenses-only contacts to not have to
input all tax details, as we may not need to invoice then, thus are
useless for us, but sometimes it might be interesting to have them,
“just in case”.
Of course, i did not want to make nullable any of the tax details
required to generate an invoice, otherwise we could allow illegal
invoices. Therefore, that data had to go in a different relation,
and invoice’s foreign key update to point to that relation, not just
customer, or we would again be able to create invalid invoices.
We replaced the contact’s trade name with just name, because we do not
need _three_ names for a contact, but we _do_ need two: the one we use
to refer to them and the business name for tax purposes.
The new contact_phone, contact_web, and contact_email relations could be
simply a nullable field, but i did not see the point, since there are
not that many instances where i need any of this data.
Now company.taxDetailsForm is no longer “the same as contactForm with
some extra fields”, because i have to add a check whether the user needs
to invoice the contact, to check that the required values are there.
I have an additional problem with the contact form when not using
JavaScript: i must set the required field to all tax details fields to
avoid the “(optional)” suffix, and because they _are_ required when
that checkbox is enabled, but i can not set them optional when the check
is unchecked. My solution for now is to ignore the form validation,
and later i will add some JavaScript that adds the validation again,
so it will work in all cases.
We have shown the application to a potential user, and they told us that
it would be very useful to have a total in the table’s footer, so that
they can verify the amount with the bank’s extracts.
It would be very unusual to have an expense from a customer, and we do
not have (yet) a name for supplier or whatever it should be here, so i
used the same name we use for the column in the table.
Had to add an `unsafe` function to be able to translate text with HTML
fragments in it, although the fragments are added back with printf
because the login link is actually not translatable.
I tried to have a log line that uses the “common” format from Apache,
because i thought that it would help me reuse regexps i have defined for
fail2ban filters and such.
However, it makes no much sense.
For once, i was repeating the date and time: log.Printf already does
that for me.
And, second, i do not need that data in Numerus’ log because i always
run it behind a proxy that _has_ a “common”-formatted log file, so
there is no need for me to repeat all that data again.
What i need is the IP, to know whether remotedAdd() function works as
expected; the method, to check that the override does its job; the path,
to know what resource the browser requested; the response status code,
so that i do not need to open the browser console for that; the response
size, to keep on eye that i do not return a lot of data; and the
total response time, to realize how long my unoptimized SQL queries
slows the application down.
The rest, Apache should do its job and record it in its log file for
fail2ban and whatever i need the logs for in the future.
I need the actual remote address to add fail2ban rules for it, but i
also to not want everyone to be able to fake X-Forward-For HTTP headers.
Which can contain multiple ip addresses, by the way, so i have to get
only the first one, as the others will be the proxies that the request
has been (re)forwarded to.
The same as for invoices: to allow people to have their own numbering
scheme, and for these that start using the program in the middle of the
current year.
I have to use a value to be used as “none” for payment method and
contact. In PL/pgSQL add_quote and edit_quote functions, that value is
NULL, while in forms it is the empty string. I can not simply pass the
empty string for either of these fields because PL/pgSQL expects
(nullable) integers, and "" is not a valid integer and is not NULL
either. A conversion is necessary.
Apparently, Go’s nil is not a valid representation for SQL’s NULL with
pgx, and had to use sql.NullString instead.
I also needed to coalesce contact’s VATIN and phone, because null values
can not be scanned to *string. I did not do that before because
`coalesce(vatin, '')` throws an error that '' is not a valid VATIN and
just left as is, wrongly expecting that pgx would do the job of leaving
the string blank for me. It does not.
Lastly, i can not blindly write Quotee’s tax details in the quote’s view
page, or we would see the (), characters for the empty address info.
This is for new users that do not start using the application from the
beginning of the current fiscal year and, therefore, need to create
invoices starting from a specific number.
I had to change the constraint on the currval to allow zero, otherwise
it would not be possible to set 1 as the next number, because users
can also not delete the row.
It is better that way because it works without JavaScript; if HTMx is
not available, it will just use regulars forms.
The problem is that most of the submit buttons where using formaction
to send the request to a different action, and only one button was the
“real” action. Since i could not pass the formaction to
invoice-product-form template, i have changed the “default” action to
the one with “ancillary” functions.
I have to use a different action to remove for each product because i
can not pass the index to the backend without JavaScript: it only
depends on the button click, that already has a name for the action.
Thus, in a way, i have “merged” the action and the index in a single
name.
There is no point in creating a new invoice without products, thus we
were forcing users to always use the “Add product” button for no reason
other than it was easier for me….
I wanted to add the product inside ServeInvoice, when the slug is “new”,
but then it tried to compute the invoice total without price or quantity
and it failed. Thus, i add that product after it has done the
computation query.
For some reason, i assumed that if the invoice product has and ID, that
is it comes from the database, it must also have a product ID, which is
incorrect, because we allow invoice lines with products not added to the
product relation.
I am using zero to mean “no product ID”, so now that validation has to
include the zero as well.
Otherwise, pgx (rightfully) tries to convert a "" into a integer, as
this is the field’s type, cannot, and panics with an error.
Added a IntegerOrNull method to FormField because this is exactly the
same that happens with the invoiceProductId, and made no sense to have
to do the logic twice, or in a function inside form.
Oriol told me what he actually wants: a way to see the current month,
quarter, and year for both double-check that the taxes form are filled
in correct and to see whether the business is doing well. This is
specially important for the quarter period, as he has to fill taxes
each quarter. Thus, the “last 90 days” thing i did was easier for me,
but completely useless for him.
We also decided to add previous month and previous quarter options
because it would be unfair to expect users check that data exactly the
last day or “lose access” to it.
I had to change MethodOverrider to check whether the form is encoded as
multipart/form-data or i would not be able to get the method field from
forms with files.
For now i add the file manually, i.e., outside add_expense and
edit_expense PL/pgSQL functions, because it was faster for me, but i
will probably add an attach_to_expense function, or something like that,
to avoid having the whole ON CONFLICT logic inside Golang—this belongs
to the database.
jordi fita mas