I need the actual remote address to add fail2ban rules for it, but i
also to not want everyone to be able to fake X-Forward-For HTTP headers.
Which can contain multiple ip addresses, by the way, so i have to get
only the first one, as the others will be the proxies that the request
has been (re)forwarded to.
The same as for invoices: to allow people to have their own numbering
scheme, and for these that start using the program in the middle of the
current year.
I have to use a value to be used as “none” for payment method and
contact. In PL/pgSQL add_quote and edit_quote functions, that value is
NULL, while in forms it is the empty string. I can not simply pass the
empty string for either of these fields because PL/pgSQL expects
(nullable) integers, and "" is not a valid integer and is not NULL
either. A conversion is necessary.
Apparently, Go’s nil is not a valid representation for SQL’s NULL with
pgx, and had to use sql.NullString instead.
I also needed to coalesce contact’s VATIN and phone, because null values
can not be scanned to *string. I did not do that before because
`coalesce(vatin, '')` throws an error that '' is not a valid VATIN and
just left as is, wrongly expecting that pgx would do the job of leaving
the string blank for me. It does not.
Lastly, i can not blindly write Quotee’s tax details in the quote’s view
page, or we would see the (), characters for the empty address info.
This is for new users that do not start using the application from the
beginning of the current fiscal year and, therefore, need to create
invoices starting from a specific number.
I had to change the constraint on the currval to allow zero, otherwise
it would not be possible to set 1 as the next number, because users
can also not delete the row.
It is better that way because it works without JavaScript; if HTMx is
not available, it will just use regulars forms.
The problem is that most of the submit buttons where using formaction
to send the request to a different action, and only one button was the
“real” action. Since i could not pass the formaction to
invoice-product-form template, i have changed the “default” action to
the one with “ancillary” functions.
I have to use a different action to remove for each product because i
can not pass the index to the backend without JavaScript: it only
depends on the button click, that already has a name for the action.
Thus, in a way, i have “merged” the action and the index in a single
name.
There is no point in creating a new invoice without products, thus we
were forcing users to always use the “Add product” button for no reason
other than it was easier for me….
I wanted to add the product inside ServeInvoice, when the slug is “new”,
but then it tried to compute the invoice total without price or quantity
and it failed. Thus, i add that product after it has done the
computation query.
For some reason, i assumed that if the invoice product has and ID, that
is it comes from the database, it must also have a product ID, which is
incorrect, because we allow invoice lines with products not added to the
product relation.
I am using zero to mean “no product ID”, so now that validation has to
include the zero as well.
Otherwise, pgx (rightfully) tries to convert a "" into a integer, as
this is the field’s type, cannot, and panics with an error.
Added a IntegerOrNull method to FormField because this is exactly the
same that happens with the invoiceProductId, and made no sense to have
to do the logic twice, or in a function inside form.
Oriol told me what he actually wants: a way to see the current month,
quarter, and year for both double-check that the taxes form are filled
in correct and to see whether the business is doing well. This is
specially important for the quarter period, as he has to fill taxes
each quarter. Thus, the “last 90 days” thing i did was easier for me,
but completely useless for him.
We also decided to add previous month and previous quarter options
because it would be unfair to expect users check that data exactly the
last day or “lose access” to it.
I had to change MethodOverrider to check whether the form is encoded as
multipart/form-data or i would not be able to get the method field from
forms with files.
For now i add the file manually, i.e., outside add_expense and
edit_expense PL/pgSQL functions, because it was faster for me, but i
will probably add an attach_to_expense function, or something like that,
to avoid having the whole ON CONFLICT logic inside Golang—this belongs
to the database.
By chance, i found out that sometimes Go returned a Content-Type header
of text/plain for some responses to HTMx request. Go’s documentation
for http.ResponseWriter.Write sheds some light to this issue:
> If the Header does not contain a Content-Type line, Write adds a
> Content-Type set to the result of passing the initial 512 bytes of
> written data to DetectContentType.
http.DetectContentType has “sniff signatures” for the most common HTML
elements, such as `<BODY`, `<P`, or even `<!--`, but when the template
only has elements not in that list, the text “text sniff signature”
kicks in because the content does not contain binary data bytes, as
specified in [0], §7.1, step 9.
I can not change mustRenderTemplate’s wr parameter, and its callers’,
to be of type http.ResponseWriter because mustWriteInvoicePdf writes
the template to a pipe object, which is not of this type. Thus, i have
to resort to type assertion inside the method.
[0]: https://mimesniff.spec.whatwg.org/#identifying-a-resource-with-an-unknown-mime-type
The product search returns a list of products using its slug as the
“external key”, because i do not want people seeing the id in links,
and the search product list is just a different rendering of the product
index table.
However, now i had two almost identical select queries for product,
one using the product_id and the other its slug, the former intended for
the form to select products using checkboxes—the one non-JavaScript
users see—and the latter for the product search.
Using the slug in both forms i can now simplify the code and have a
single query.
With Oriol agreed that adding or editing invoices, products, and
contacts is not just a “user interruption” but the main flow of the
program, and, as such, it is not correct to use dialogs for these.
More importantly, it was harder to concentrate, specially with the more
involved form of invoices, because of all the “noise” behind the dialog.
This is for “free products”, where the user adds an invoice row, does
not select a product from the search, and clicks the update button.
Numerus should select “appropriate” values for those that are left
unspecified.
I also no longer require the product_id to be an integer; if it is
empty, then it is assumed to be a “free product”.
I actually find more comfortable to select the product from the list
presented up until now, but this is mostly because i have very few
products and the list is not too long, so the idea is that with
JavaScript we will dynamically add an empty product row to the invoice
and then use the name field to search the product by name.
I have the feeling that i am doing something wrong because i ended up
with a lot of HTMx attribute for what i feel is not that much work,
but for now it will work.
I have added the `Is` field to `InputField` in order to include the `id`
attribute to the HTML element, because the HTMLAttributes are attached
to the `input`, not the `div`, and i felt like this one should also be
a custom element based on div, like all the others.
These is not yet any keyboard control to select the search results.
I am not happy with having the search of products in a different URL
than the index, specially since they use the exact same SQL query and
ProductFilter struct, but i did not know how else ask for a different
representation without resorting to the more complicated MIME types.
I need a way to search products by name in the invoice form, when the
user adds or changes a product. Since this is something that i have to
add too to the product list, i added it now so the function will already
be ready.
We are going to allow invoices with products that are not (yet) inserted
into the products table.
We always allowed to have products in invoices with a totally different
name, description, price, and whatnot, but until now we had the product
id in these invoice lines for statistics purposes.
However, Oriol raised the concern that this requires for the products
to be inserted before we can create an invoice with them, and we do not
plan to have a “create product while invoicing” feature, thus it would
mean that people would need to cancel the new invoice, create the new
product, and then start the invoice again from scratch.
The compromise is to allow products in the invoice that do not have a
product_id, meaning that at the time the invoice was created they were
not (yet) in the products table. Oriol sees this stop-invoice-create-
product issue more important than the accurate statistics of product
sales, as it will probably be only one or two units off, anyway.
I did not want to allow NULL values to the invoice product’s product_id
field, because NULL means “dunno” instead of “no product”, so i had to
split that field to a separate table that relates an invoice product
with a registered product.
We have reconsidered the toggle thing and instead moved the selection
into a little menu on top of the input, like the input’s label does à
la Material Design.
I just moved the checkboxes into a new details, that works as a menu,
but i had to add the type="search" to the existing input in the tags
field, or the CSS would style the checkboxes as well.
I do not do anything when the checkbox selection changes because that
already triggers a POST to the server that returns the new HTML with
the checkbox changed, and the JavaScript only has to retrieve that new
structure, exactly as it does in the initial rendering.
Since we want to add a little description to the options, i no longer
can use the same SelectOption in ToggleField, even though i could have
reused the Group element, but that felt wrong.
I realized that using a select for just two, short, options is overkill:
the select and its options use a lot more real state than the two
radios, which can have tooltips (not yet, though).
Since i am going to replace this field with a custom element that has
a toggle-like aspect, i already added the is="numerus-toggle" attribute
and use it for stying the non-JavaScript field.
This is because Oriol thinks that there may be cases where you want to
search invoices and such that have any of the selected labels, not all
of them, so we agreed on adding an option to choose.
The idea is that it will be a toggle, but this requires JavaScript and
this commit adds it as a dropdown as a first non-JavaScript step.
I use the same pattern as HTMx’s “Click to Edit” example[0], except that
my edit form is triggered by submit and by focus out of the tags input.
I could not, however, use the standard focus out event because it would
also trigger when removing a tag with the mouse, as for a moment the
remove button has the focus and the search input dispatches a bubbling
focusout. I had to resort to a custom event for that, but i am not
happy with it.
The autofocus attribute seems to do nothing in this case, so i need to
manually change the focus to the new input with JavaScript. However,
this means that i can not use the same input ID for all the forms
because getElementById would always return the first in document order,
changing the focus to that same element and automatically submit the
form due to focus out. That’s why in this form i append the invoice’s
slug to the input’s ID.
Finally, this is the first time i am using an HTMx-only solution and i
needed a way to return back just the HTML for the <td>, without <title>,
breadcrumbs, or <dialog>. In principle, the template would be the
“layout”, but then i would need to modify everything to check whether
the template file is empty, or something to that effect, so instead i
created a “standalone” template for these cases.
[0]: https://htmx.org/examples/click-to-edit/
It all started when i wanted to try to filter invoices by multiple tags
using an “AND”, instead of “OR” as it was doing until now. But
something felt off and seemed to me that i was doing thing much more
complex than needed, all to be able to list the tags as a suggestion
in the input field—which i am not doing yet.
I found this article series[0] exploring different approaches for
tagging, which includes the one i was using, and comparing their
performance. I have not actually tested it, but it seems that i have
chosen the worst option, in both query time and storage.
I attempted to try using an array attribute to each table, which is more
or less the same they did in the articles but without using a separate
relation for tags, and i found out that all the queries were way easier
to write, and needed two joins less, so it was a no-brainer.
[0]: http://www.databasesoup.com/2015/01/tag-all-things.html
This makes reload only the <main> portion of the page, instead of the
whole thing, which to me looks faster; haven’t really measured it.
Like with duplicate, i had to add the location query argument when
inside the view page in order to return back to the same page, not the
index.
Had to add a new hidden field to the form to know whether, when the
request is HTMx-triggered, to refresh the page, as i do when duplicating
from the index, or redirect the client to the new invoice’s view page,
but only if i was duplicating from that same page, not the index.
Since i now have to target main when redirecting to the view page, so
i had to add a location structure with the required json fields and all
that, when “refreshing” i actually tell HTMx to open the index page
again, which seems faster, now that i am used to boosted links.
Changed the invoice number field’s type to search to add the delete icon
on Chromium. Firefox does not add that icon, but i do not care; it is
still better that type="text".
Had to emit the change event to the numerus-tag field, otherwise the
form would not detect the change.
I also can not use keyup as a trigger because the changed modifier can
not be used in the <form>, as nothing ever changes, i do not know how to
trigger the form from children (i.e., data-hx-trigger on the <input>
does nothing), and i can not trigger for just any keyup, or i would
make the request even if they only moved the cursor with the arrow keys,
which is very confusing as Firefox resets the position (this may be due
the fact that i reload the whole <main>, but still).
This is for the unusual case of products without taxes, that PostgreSQL
by default would generate an array with a single null value in it, but
then pgx would not be able to set that null to the string variable.
Initially, this field was meant to be left almost always blank, except
for when we deleted invoiced and had to “replace” its number with a new
invoice; using the automatic numbering in this cas would not “fill in”
the missing number in the sequence.
However, we decide to not allow removing invoicer not edit their
numbers, therefore, if everything goes as planned, there should not be
any gap in the sequence, and that field is rendered useless.
Oriol suggested making it a read-only field, both for new and edit
forms, but i do not think it makes sense to have a field if you can not
edit it at all, specially in the new invoice dialog, where it would
always be blank. In the edit form we already show the number in the
title and breadcrumbs, thus no need for the read-only field as
reference.
I still keep a Number member to the form struct, but is now a string
(kind of “a read-only field”, in a way) and just to be written in the
title or breadcrumbs. I did not like the idea of adding a new SQL
query just for that value.
In this case i have to use the same id for the dialog content in all
pages because, for now, there are a couple of forms that need to replace
it on submit—the new/edit form and the product selection form.
Unfortunately, HTMx does not have support for `formaction` attribute at
this point, so i had to use the workaround described in [0].
[0] https://github.com/bigskysoftware/htmx/issues/623
Instead of using links in the invoice tags, that we will replace with a
“click-to-edit field”, with Oriol agreed to add a form with filters that
includes not only the tags but also dates, customer, status, and the
invoice number.
This means i now need dynamic SQL, and i do not think this belongs to
the database (i.e., no PL/pgSQL function for that). I have looked at
query builder libraries for Golang, and did not find anything that
suited me: either they wanted to manage not only the SQL query but also
all structs, or they managed to confuse Goland’s SQL analyzer.
For now, at least, i am using a very simple approach with arrays, that
still confuses Goland’s analyzer, but just in a very specific part,
which i find tolerable—not that their analyzer is that great to begin
with, but that’s a story for another day.
We agreed with Oriol that this link would only serve to confuse people.
I initially added the link because i thought it was a shame to have to
navigate to the contact section to look or change the info of a customer
that you have an invoice for in front of you. However, it makes little
sense to be able to edit the contact from both sections, and we do not
have a “view page” for contacts to link to, thus the removal.
Had to add the editProductPage because now i need to know the slug in
order to build the form’s action link. I also added the `ProductName`
field because it was less awkward than using `.Form.Name` everywhere.
With Oriol we agreed that products should have tags, too, and that the
“tag pool”, as it were, should be shared with the one for invoices and
contacts.
Had to add the `company_id` attribute in the `using` clause for `tag` in
`MustFillFromDatabase`, even though it’s not strictly necessary, because
then PostgreSQL does not know which `company_id` attribute use for the
join with `company`—the one from `product` or the one from `tag`.
With Oriol we agreed that contacts should have tags, too, and that the
“tag pool”, as it were, should be shared with the one for invoices (and
all future tags we might add).
I added the contact_tag relation and tag_contact function, just like
with invoices, and then realized that the SQL queries that Go had to
execute were becoming “complex” enough: i had to get not only the slug,
but the contact id to call tag_contact, and all inside a transaction.
Therefore, i opted to create the add_contact and edit_contact functions,
that mirror those for invoice and products, so now each “major” section
has these functions. They also simplified a bit the handling of the
VATIN and phone numbers, because it is now encapsuled inside the
PL/pgSQL function and Go does not know how to assemble the parts.
I am not sure if, at the end, all pages that now use
mustRenderAppTemplate will be replaced with mustRenderMainTemplate,
but for now i keep them separate to know which routes are already
“boosted”.
Had to change the data context for that template to include the Slug,
so that the <form> element can set the correct `action` instead of
using the current URI, as it is no longer “correct” (form-wise) when
using HTMx.
In that case, strings.Split() return an array with a single empty string
element, that does not pass the domain check for tag_name in the
database.
And an invoice with no tags would get an array of a single NULL in
array_agg, so i had to convert it to an empty string in order for it
to work as expected.
I had to change the way /invoices/new and /invoices/batch are handled,
because httprouter was not happy with the new POST /invoices/:slug/edit
route, claiming that /invoices/:slug conflicts with the previously
existing routes.
I also could not make it work with the PATCH method, even though i
correctly added the patchMethod override function, therefore editing
invoices is also weird because i have to take into account the “quick”
invoice status change.
I use the same form for both new and edit invoices, because the only
changes are that we can not edit the invoice date and number, by
Oriol’s design, but must be able to change the status; very similar
forms.
I had to use a deferrable foreign key because the payment methods have
a reference to the company, and the company now a circular reference to
payment method.
This was actually the (first) reason we added the tax classes: to show
them in columns on the invoice—without the class we would need a column
for each tax rate, even though they are the same tax.
The invoice design has the product total with taxes at the last column,
above the tax base, that i am not so sure about, but it seems that it
has not brought any problem whatsoever so far, so it remains as is.
Had to reduce the invoice’s font size to give more space to the table
or the columns would be right next to each other. Oriol also told me
to add more vertical spacing to the table’s footer.
We only allow a single tax of each class in products and invoices,
because it does not make sense to add two different VAT to the same
product, for instance.
We will only allow to select a tax from each of the tax classes, but
the user needs to know what class each tax belongs to, and grouping
the taxes by class in the select helps with that.
We want to show the percentage of the tax as columns in the invoice,
but until now it was not possible to have a single VAT column when
products have different VAT (e.g., 4 % and 10 %), because, as far
as the application is concerned, these where ”different taxes”. We
also think it would be hard later on to compute the tax due to the
government.
So, tax classes is just a taxonomy to be able to have different names
and rates for the same type of tax, mostly VAT and retention in our
case.
Apparently, in older version of WeasyPrint it was possible to output
either PNG or PDF, so they had that parameter. In more recent versions,
the only acceptable is PDF, but they removed the fucking parameter
between versions 51 and 52, to it _will_ fail in recent version of
WeasyPrint.
For now, i am using the same version that Debian 11 does, and let’s
hope it stays like this a long time. (It won’t, of course, but well….)
Although it is possible to just print the invoice from the browser, many
people will not even try an assume that they can not create a PDF for
the invoice.
I thought of using Groff or TeX to create the PDF, but it would mean
maintaining two templates in two different systems (HTML and whatever i
would use), and would probably look very different, because i do not
know Groff or TeX that well.
I wish there was a way to tell the browser to print to PDF, and it can
be done, but only with the Chrome Protocol to a server-side running
Chrome instance. This works, but i would need a Chrome running as a
daemon.
I also wrote a Qt application that uses QWebEngine to print the PDF,
much like wkhtmltopdf, but with support for more recent HTML and CSS
standards. Unfortunately, Qt 6.4’s embedded Chromium does not follow
break-page-inside as well as WeasyPrint does.
To use WeasyPrint, at first i wanted to reach the same URL as the user,
passing the cookie to WeasyPrint so that i can access the same invoice
as the user, something that can be done with wkhtmltopdf, but WeasyPrint
does not have such option. I did it with a custom Python script, but
then i need to package and install that script, that is not that much
work, but using the Debian-provided script is even less work, and less
likely to drift when WeasyPrint changes API.
Also, it is unnecessary to do a network round-trip from Go to Python
back to Go, because i can already write the invoice HTML as is to
WeasyPrint’s stdin.
jordi fita mas