We are going to allow invoices with products that are not (yet) inserted
into the products table.
We always allowed to have products in invoices with a totally different
name, description, price, and whatnot, but until now we had the product
id in these invoice lines for statistics purposes.
However, Oriol raised the concern that this requires for the products
to be inserted before we can create an invoice with them, and we do not
plan to have a “create product while invoicing” feature, thus it would
mean that people would need to cancel the new invoice, create the new
product, and then start the invoice again from scratch.
The compromise is to allow products in the invoice that do not have a
product_id, meaning that at the time the invoice was created they were
not (yet) in the products table. Oriol sees this stop-invoice-create-
product issue more important than the accurate statistics of product
sales, as it will probably be only one or two units off, anyway.
I did not want to allow NULL values to the invoice product’s product_id
field, because NULL means “dunno” instead of “no product”, so i had to
split that field to a separate table that relates an invoice product
with a registered product.
It all started when i wanted to try to filter invoices by multiple tags
using an “AND”, instead of “OR” as it was doing until now. But
something felt off and seemed to me that i was doing thing much more
complex than needed, all to be able to list the tags as a suggestion
in the input field—which i am not doing yet.
I found this article series[0] exploring different approaches for
tagging, which includes the one i was using, and comparing their
performance. I have not actually tested it, but it seems that i have
chosen the worst option, in both query time and storage.
I attempted to try using an array attribute to each table, which is more
or less the same they did in the articles but without using a separate
relation for tags, and i found out that all the queries were way easier
to write, and needed two joins less, so it was a no-brainer.
[0]: http://www.databasesoup.com/2015/01/tag-all-things.html
Initially, this field was meant to be left almost always blank, except
for when we deleted invoiced and had to “replace” its number with a new
invoice; using the automatic numbering in this cas would not “fill in”
the missing number in the sequence.
However, we decide to not allow removing invoicer not edit their
numbers, therefore, if everything goes as planned, there should not be
any gap in the sequence, and that field is rendered useless.
Oriol suggested making it a read-only field, both for new and edit
forms, but i do not think it makes sense to have a field if you can not
edit it at all, specially in the new invoice dialog, where it would
always be blank. In the edit form we already show the number in the
title and breadcrumbs, thus no need for the read-only field as
reference.
I still keep a Number member to the form struct, but is now a string
(kind of “a read-only field”, in a way) and just to be written in the
title or breadcrumbs. I did not like the idea of adding a new SQL
query just for that value.
With Oriol we agreed that products should have tags, too, and that the
“tag pool”, as it were, should be shared with the one for invoices and
contacts.
Had to add the `company_id` attribute in the `using` clause for `tag` in
`MustFillFromDatabase`, even though it’s not strictly necessary, because
then PostgreSQL does not know which `company_id` attribute use for the
join with `company`—the one from `product` or the one from `tag`.
With Oriol we agreed that contacts should have tags, too, and that the
“tag pool”, as it were, should be shared with the one for invoices (and
all future tags we might add).
I added the contact_tag relation and tag_contact function, just like
with invoices, and then realized that the SQL queries that Go had to
execute were becoming “complex” enough: i had to get not only the slug,
but the contact id to call tag_contact, and all inside a transaction.
Therefore, i opted to create the add_contact and edit_contact functions,
that mirror those for invoice and products, so now each “major” section
has these functions. They also simplified a bit the handling of the
VATIN and phone numbers, because it is now encapsuled inside the
PL/pgSQL function and Go does not know how to assemble the parts.
We plan to tag also contacts and products using the same tag relation,
but different invoice_tag, contact_tag, and product_tag relations for
each one. However, the logic is the same for all three, hence it makes
more sense to put it into a PL/pgSQL with dynamic SQL. Moreover, the
SQL for tagging in add_invoice and edit_invoice where almost exactly
the same, the only difference was deleting the existing tags when
editing.
I do not execute the tag_relation function in its test suite because
by itself it does nothing without supporting invoice_tag, contact_tag,
or any such relation, so it is being tested in the suite for
tag_invoice.
I had to use a deferrable foreign key because the payment methods have
a reference to the company, and the company now a circular reference to
payment method.
This was actually the (first) reason we added the tax classes: to show
them in columns on the invoice—without the class we would need a column
for each tax rate, even though they are the same tax.
The invoice design has the product total with taxes at the last column,
above the tax base, that i am not so sure about, but it seems that it
has not brought any problem whatsoever so far, so it remains as is.
Had to reduce the invoice’s font size to give more space to the table
or the columns would be right next to each other. Oriol also told me
to add more vertical spacing to the table’s footer.
We want to show the percentage of the tax as columns in the invoice,
but until now it was not possible to have a single VAT column when
products have different VAT (e.g., 4 % and 10 %), because, as far
as the application is concerned, these where ”different taxes”. We
also think it would be hard later on to compute the tax due to the
government.
So, tax classes is just a taxonomy to be able to have different names
and rates for the same type of tax, mostly VAT and retention in our
case.
The design calls for rendering all amounts with their currency symbol,
but golang.org/x/text’s currency package always render the symbol in
front, which is wrong in Catalan and Spanish, and a lot of other
languages.
Consulting the Internet, the most popular package for that is
accounting[0], which is almost as useless because they confuse locale
with the currency’s country of origin’s “usual locale” (e.g., en-US for
USD), which is also wrong: in Catalan i need to write USD prices as
"1.234,56 $" regardless of what Americans do.
With accounting i have the recourse of initializing the struct that
holds all the “locale” information, which is also wrong because i have
to define the decimal and thousands separators, something that depends
only on the locale, next to the currency’s precision, that is
locale-independent. But, since all CLDR data from golang.org/x/text
is inside an internal package, i can not access it and would need to
define all that information myself, which defeats the purpose of using
an external package.
Since for now i only need the format pattern for currency, i just saved
it into the database of available languages, that i do not expect to
grow too much.
[0]: https://github.com/leekchan/accounting
They are not functions because i need to join them with the main
invoice relation, and although possible is a bit more awkward with
functions.
The taxes have their own relation because i will need them grouped by
their name in the PDF, so it will probably be a select for that
relation.
I can not use a PostgreSQL sequence because invoices need to be gapless,
and sequences are designed to not rollback, for performance reasons. In
this case, the performance is secondary because the law does not care.
I am going to add similar functions for invoices, as i will need to
add the taxes for their products and their own taxes, thus the Go code
will begin to be “too much work” and i feel better if that is in
PL/pgSQL.
If i have these functions for invoices, there is no point on having to
do almost the same work, albeit less, for products.
I store again the product’s name, description, and prices, because they
are bound to change, but the invoice should remain the same always.
That makes me wonder if i should do the same for seller’s and buyer’s
data, but that should be a different commit.
I’ve added the discount_rate domain because then i can test it
independently of the invoice_product relation, moreover i am sure i will
need it for simplified invoices too.
It seems that we do not agree en whether the IRPF tax should be
something of the product or the contact, so we decided to make the
product have multiple taxes, just in case, and if only one is needed,
then users can just select one; no need to limit to one.
I do not want to use floats in the Go lang application, because it is
not supposed to do anything with these values other than to print and
retrieve them from the user; all computations will be performed by
PostgreSQL in cents.
That means i have to “convert” from the price format that users expect
to see (e.g., 1.234,56) to cents (e.g., 123456) and back when passing
data between Go and PostgreSQL, and that conversion depends on the
currency’s decimal places.
At first i did everything in Go, but saw that i would need to do it in
a loop when retrieving the list of products, and immediately knew it was
a mistake—i needed a PL/pgSQL function for that.
I still need to convert from string to float, however, when printing the
value to the user. Because the string representation is in C, but i
need to format it according to the locale with golang/x/text. That
package has the information of how to correctly format numbers, but it
is in an internal package that i can not use, and numbers.Digit only
accepts numeric types, not a string.
At first we thought that a regular text field would do, because we were
afraid that a dropdown would be worse from the point of view of user
experience, but then we realized that we need the country code for VAT
and phone validation, and we can not expect users to input that, of
course.
I had to add the first “i18n table” to the database with the name of all
countries in both Catalan and Spanish and Catalan; English is the
default. For now i think i do not need a view that would select the
name based on the locale of the current request, because currently i do
not plan on adding any other such table —the currency uses the code and
the symbol, thus no need for localization.
However, now i need the language tag from the locale in order to get the
correct translation, and gotext does not give me any way to access the
inner language. Thus the need for our Locale type.
I do not have more time to update the update to the company today, but i
believe this is already a good amount of work for a commit.
The company is going to be used for row level security, as users will
only have access to the data from companies they are granted access, by
virtue of being in the company_user relation.
I did not know how add a row level security policy to the company_user
because i needed the to select on the same relation and this is not
allowed, because it would create an infinite loop.
Had to add the vat, pg_libphonenumber, and uri extensions in order to
validate VAT identification numbers, phone numbers, and URIs,
repectively. These libraries are not in Debian, but i created packages
for them all in https://dev.tandem.ws/tandem.
This function does not ask for the confirmation because this is an
user-facing issue, not for the database.
Still missing: validation and proper error messages.
This is for security, just in case two users have the same cookie,
althought it is unlikely, but nevertheless less guessable.
I also need to refresh the cookie when the user changes their email
address, because it is liked toghether. It does mean that it will
logout from everywhere else, but i can not do anything about that.
Since users do not have access to the auth scheme, i had to add a view
that selects only the data that they can see of themselves (i.e., no
password or cookie).
I wanted to use the `request.user.id` setting that i set in
check_cookie, but this would be bad because anyone can change that
parameter and, since the view is created by the owner, could see and
*change* the values of everyone just by knowing their id. Thus, now i
use the cookie instead, because it is way harder to figure out, and if
you already have it you can just set to your browser and the user is
fucked anyway; the database can not help here.
I **am** going to use the user id in row level security policies, but
not the value coming for the setting but instaed the one in the
`user_profile`, since it already is “derived” from the cookie, that’s
why i added that column to the view.
The profile includes the language, that i do not use it yet to switch
the locale, so i had to add a relation of the available languages, for
constraint purposes. There is no NULL language, and instead i added the
“Undefined” language, with ‘und’ tag’, to represent “do not know/use
content negotiation”.
The languages in that relation are the same i used to have inside
locale.go, because there is no point on having options for languages i
do not have the translation for, so i now configure the list of
available languages user in content negotiation from that relation.
Finally, i have added all font from RemixIcon because that’s what we
used in the design and i am going to use quite a lot of them.
There is duplication in the views; i will address that in a different
commit.
I did not like the idea that it was the Go server who should set values
such as request.user or set the role, because this is mostly something
that only the database wants for itself, such as when calling logout. I
am also planning to use these setings for row security with the user’s
id, that the Go application has no need for, but with the current
approach i would need to return it from check_cookie so that it can
return it back to the database when acquiring the connection.
I would have used the same function to set the settings and the role,
but security definer functions—obviously in retrospect—can not set the
role, because then could switch to any role of the user that defined the
function, not the roles they are member of. Thus, a new function.
I did not want to do that every time i needed the database connection
within the same request, because it would perform the same operations
each time—it is the same cookie, afterall—, so new connections are
request scoped and passed along in the context.
I do not want to give access to authenticator until i know who the user
is, herefore that function can not be in the numerus schema as the
authenticator user can not see it.
At first i thought that i would need to implement sessions, the ones
that keep small files onto the disk, to know which user is talking to
the server, but then i realized that, for now at least, i only need a
very large number, plus the email address, to be used as a lookup, and
that can be stored in the user table, in a separate schema.
Had to change login to avoid raising exceptions when login failed
because i now keep a record of login attemps, and functions are always
run in a single transaction, thus the exception would prevent me to
insert into login_attempt. Even if i use a separate procedure, i could
not keep the records.
I did not want to add a parameter to the logout function because i was
afraid that it could be called from separate users. I do not know
whether it is possible with the current approach, since the settings
variable is also set by the same applications; time will tell.
I honestly do not remember why i thought i needed the find_user
function: it is just a select with a query that i only need in a single
place—when login.
I belive it was a missguided attempt to “write the function safer”, in
hopes that calling a function won’t have the same problems as when
querying a table, but this is fixed with the search_path, that i added.
There is no pgTAP for this, i believe.
User authentication is based on PostgREST’s[0]: There is a noninherit
role, authenticator, whose function is only to switch to a different
role according to the application’s session. Accordingly, this role has
no permission for anything.
The roles that this authentication can switch to are guest, invoicer, or
admin. Guest is for anonymous users, when they need to login or
register; invoicers are regular users; and admin are application’s
administrators, that can change other user’s status, when they have to
be removed or have they password changed, for example.
The user relation is actually inaccessible to all roles and can only be
used through a security definer function, login, so that passwords are
not accessible from the application.
I hesitated on what to use as the user’s primary key. The email seemed
a good candiate, because it will be used for login. But something rubs
me the wrong way.
It is not that they can change because, despite what people on the
Internet keeps parroting, they do not need to be “immutable”, PostgreSQL
can cascade updates to foreign keys, and people do **not** change email
addresses that ofter.
What i **do** know is that email addresses should be unique in order to
be used for login and password, hovewer i had to decide what “unique”
means here, because the domain part is case insensitive, but the local
part who knows? I made the arbitrary decision of assuming that the
whole address is case sensitive.
I have the feeling that this will bite me harder in the ass than using
it as the primary key.
[0]: https://postgrest.org/en/stable/auth.html
jordi fita mas