When i wrote the functions to import contact, i already created a couple
of “temporary” functions to validate whether the input given from the
Excel files was correct according to the various domains used in the
relations, so i can know whether i can import that data.
I realized that i could do exactly the same when validating forms: check
that the value conforms to the domain, in the exact same way, so i can
make sure that the value will be accepted without duplicating the logic,
at the expense of a call to the database.
In an ideal world, i would use pg_input_is_valid, but this function is
only available in PostgreSQL 16 and Debian 12 uses PostgreSQL 15.
These functions are in the public schema because initially i wanted to
use them to also validate email, which is needed in the login form, but
then i recanted and kept the same email validation in Go, because
something felt off about using the database for that particular form,
but i do not know why.
Depending on the number of test ran before this test, the sequence could
overlap with the existing ids and fail because the on conflict do update
would update multiple rows.
This allows to import an Excel file exported from Holded, because it is
our own user case. When we have more customers, we will give out an
Excel template file to fill out.
Why XLSX files instead of CSV, for instance? First, because this is the
output from Holded, but even then we would have more trouble with CSV
than with XLSX because of Microsoft: they royally fucked up
interoperability when decided that CSV files, the files that only other
applications or programmers see, should be “localized”, and use a comma
or a **semicolon** to separate a **comma** separated file depending on
the locale’s decimal separator.
This is ridiculous because it means that CSV files created with an Excel
in USA uses comma while the same Excel but with a French locale expects
the fields to be separated by semicolon. And for no good reason,
either.
Since they fucked up so bad, decided to add a non-standard “meta” field
to specify the separator, writing a `sep=,` in the first line, but this
only works for reading, because saving the same file changes the
separator back to the locale-dependent character and removes the “meta”
field.
And since everyone expects to open spreadsheet with Excel, i can not
use CSV if i do not want a bunch of support tickets telling me that the
template is all in a single line.
I use an extremely old version of a xlsx reading library for golang[0]
because it is already available in Debian repositories, and the only
thing i want from it is to convert the convoluted XML file into a
string array.
Go is only responsible to read the file and dump its contents into a
temporary table, so that it can execute the PL/pgSQL function that will
actually move that data to the correct relations, much like add_contact
does but in batch.
In PostgreSQL version 16 they added a pg_input_is_valid function that
i would use to test whether input values really conform to domains,
but i will have to wait for Debian to pick up the new version.
Meanwhile, i use a couple of temporary functions, in lieu of nested
functions support in PostgreSQL.
Part of #45
[0]: https://github.com/tealeg/xlsx
Eventually, we will allow people to upload their own profile images,
but until then we’ll use a “generic” profile icon instead of the closed
icon, that means nothing to users.
We wanted to have the same icon for that menu than for the Account item,
but the one we used until now did not look inside the round button for
the menu, thus we changed the two.
Related to #55.
These two fields are just for information purposes, as Numerus does not
have any way to wire transfer using these, but people might want to keep
these in the contact’s info as a convenience.
Since not every contact should have an IBAN, e.g., customers, and inside
SEPA (European Union and some more countries) the BIC is not required,
they are in two different relations in order to be optional without
using NULL.
For the IBAN i found an already made PostgreSQL module, but for BIC i
had to write a regular expression based on the information i gathered
from Wikipedia, because the ISO standard is not free.
These two parameters for the add_contact and edit_contact functions are
TEXT because i realized that these functions are intended to be used
from the web application, that only deals with texts, so the
ValueOrNil() function was unnecessarily complex and PostreSQL’s
functions were better suited to “convert” from TEXT to IBAN or BIC.
The same is true for EMAIL and URI domains, so i changed their parameter
types to TEXT too.
Closes#54.
We need to have contacts with just a name: we need to assign
freelancer’s quote as expense linked the government, but of course we
do not have a phone or email for that “contact”, much less a VATIN or
other tax details.
It is also interesting for other expenses-only contacts to not have to
input all tax details, as we may not need to invoice then, thus are
useless for us, but sometimes it might be interesting to have them,
“just in case”.
Of course, i did not want to make nullable any of the tax details
required to generate an invoice, otherwise we could allow illegal
invoices. Therefore, that data had to go in a different relation,
and invoice’s foreign key update to point to that relation, not just
customer, or we would again be able to create invalid invoices.
We replaced the contact’s trade name with just name, because we do not
need _three_ names for a contact, but we _do_ need two: the one we use
to refer to them and the business name for tax purposes.
The new contact_phone, contact_web, and contact_email relations could be
simply a nullable field, but i did not see the point, since there are
not that many instances where i need any of this data.
Now company.taxDetailsForm is no longer “the same as contactForm with
some extra fields”, because i have to add a check whether the user needs
to invoice the contact, to check that the required values are there.
I have an additional problem with the contact form when not using
JavaScript: i must set the required field to all tax details fields to
avoid the “(optional)” suffix, and because they _are_ required when
that checkbox is enabled, but i can not set them optional when the check
is unchecked. My solution for now is to ignore the form validation,
and later i will add some JavaScript that adds the validation again,
so it will work in all cases.
This is to avoid the problem that if i mistype `company_id = 1` instead
of `company_id = $1`, like i fixed in ee0b5d0b, at least it won’t work
in testing now, as all IDs have a three digits number.
We have shown the application to a potential user, and they told us that
it would be very useful to have a total in the table’s footer, so that
they can verify the amount with the bank’s extracts.
It would be very unusual to have an expense from a customer, and we do
not have (yet) a name for supplier or whatever it should be here, so i
used the same name we use for the column in the table.
Had to add an `unsafe` function to be able to translate text with HTML
fragments in it, although the fragments are added back with printf
because the login link is actually not translatable.
It turns out i have been **years** doing this wrong: you are supposed to
pass that value as a text, like 'CURRENT_TIMESTAMP', not like the
keyword so that it returns the current timestamp as a timestamptz.
However, i have been doing it wrong because of a bug in previous
versions of pgTAP[0], that did not take into account keywords such as
CURRENT_TIMESTAMP or CURRENT_DATE and was comparing their actual values,
not the names, therefore i thought that i misread the documentation.
Only now have discovered this because Debian 12 upgraded pgTAP version
to 1.2.0.
[0]: https://github.com/theory/pgtap/issues/244
I tried to have a log line that uses the “common” format from Apache,
because i thought that it would help me reuse regexps i have defined for
fail2ban filters and such.
However, it makes no much sense.
For once, i was repeating the date and time: log.Printf already does
that for me.
And, second, i do not need that data in Numerus’ log because i always
run it behind a proxy that _has_ a “common”-formatted log file, so
there is no need for me to repeat all that data again.
What i need is the IP, to know whether remotedAdd() function works as
expected; the method, to check that the override does its job; the path,
to know what resource the browser requested; the response status code,
so that i do not need to open the browser console for that; the response
size, to keep on eye that i do not return a lot of data; and the
total response time, to realize how long my unoptimized SQL queries
slows the application down.
The rest, Apache should do its job and record it in its log file for
fail2ban and whatever i need the logs for in the future.
I need the actual remote address to add fail2ban rules for it, but i
also to not want everyone to be able to fake X-Forward-For HTTP headers.
Which can contain multiple ip addresses, by the way, so i have to get
only the first one, as the others will be the proxies that the request
has been (re)forwarded to.
The same as for invoices: to allow people to have their own numbering
scheme, and for these that start using the program in the middle of the
current year.
I have to use a value to be used as “none” for payment method and
contact. In PL/pgSQL add_quote and edit_quote functions, that value is
NULL, while in forms it is the empty string. I can not simply pass the
empty string for either of these fields because PL/pgSQL expects
(nullable) integers, and "" is not a valid integer and is not NULL
either. A conversion is necessary.
Apparently, Go’s nil is not a valid representation for SQL’s NULL with
pgx, and had to use sql.NullString instead.
I also needed to coalesce contact’s VATIN and phone, because null values
can not be scanned to *string. I did not do that before because
`coalesce(vatin, '')` throws an error that '' is not a valid VATIN and
just left as is, wrongly expecting that pgx would do the job of leaving
the string blank for me. It does not.
Lastly, i can not blindly write Quotee’s tax details in the quote’s view
page, or we would see the (), characters for the empty address info.
They are mostly the same as invoices, but the contact and payment method
are optional, thus, like other optionals fields, i created relations to
link these that have payment method or contact, to avoid NULL columns in
quote.
Still missing functions to add and edit quotations, and views to compute
their tax and total amount.
This is because when i create the demo users then i can no remove the
available languages before users, due the constrain, and i can no use
sqitch rebase or revert.
This is for new users that do not start using the application from the
beginning of the current fiscal year and, therefore, need to create
invoices starting from a specific number.
I had to change the constraint on the currval to allow zero, otherwise
it would not be possible to set 1 as the next number, because users
can also not delete the row.
It is better that way because it works without JavaScript; if HTMx is
not available, it will just use regulars forms.
The problem is that most of the submit buttons where using formaction
to send the request to a different action, and only one button was the
“real” action. Since i could not pass the formaction to
invoice-product-form template, i have changed the “default” action to
the one with “ancillary” functions.
I have to use a different action to remove for each product because i
can not pass the index to the backend without JavaScript: it only
depends on the button click, that already has a name for the action.
Thus, in a way, i have “merged” the action and the index in a single
name.
jordi fita mas
oriol carbonell pujolàs