Question 1

How do you calculate the number of parameters in a neural network layer?

Accepted Answer

Each layer type has its own formula. A Linear layer has input_dim x output_dim weights plus output_dim biases. A Conv2D layer has in_channels x out_channels x kernel_size squared weights plus out_channels biases. An Embedding layer has vocab_size x embedding_dim weights. An LSTM layer has 4 x (input_size x hidden_size + hidden_size squared + 2 x hidden_size) parameters per layer per direction, and a GRU has the same formula with 3 gates instead of 4. Activation, dropout and pooling layers have no learnable parameters.

Question 2

How much GPU memory does training a model need?

Accepted Answer

With the Adam optimizer, budget roughly 16 bytes per parameter. That figure does not change when you switch to mixed precision. Pure FP32 training holds 4 bytes of weights, 4 of gradients and 8 of optimizer moments. Mixed precision holds 2 bytes of weights and 2 of gradients but adds an FP32 master copy of the weights, so the total is the same 16 bytes. Activation memory is separate and depends on batch size and input shape.

Question 3

Does FP16 or BF16 halve training memory?

Accepted Answer

No. Mixed precision halves the memory used for weights and gradients, but the optimizer still keeps an FP32 master copy of the weights alongside its two moment buffers. Those FP32 states cancel out the saving, so Adam training costs about 16 bytes per parameter at any weight precision. Inference is different: there, FP16 and BF16 genuinely halve the memory compared to FP32, and INT8 quarters it.

Question 4

How many parameters does a transformer block have?

Accepted Answer

A standard encoder block has 4 x (d_model squared + d_model) parameters for the query, key, value and output projections, plus 2 x d_model x d_ff + d_ff + d_model for the feed-forward network, plus 4 x d_model for the two layer norms. When d_ff is 4 x d_model, that simplifies to 12 x d_model squared + 13 x d_model. A block with d_model 512 and d_ff 2048 therefore has 3,152,384 parameters.

Question 5

How do you count FLOPs for a forward pass?

Accepted Answer

Count a multiply-accumulate as two floating point operations. A Linear layer costs 2 x input_dim x output_dim. A Conv2D layer costs 2 x in_channels x out_channels x kernel_size squared x output_height x output_width. Attention costs 2 x seq_len squared x d_model for the score matrix plus 8 x seq_len x d_model squared for the four projections. FLOPs depend on input shape, so any figure is only meaningful alongside the assumed shape.

Question 6

Is LayerCal free, and does it send my data anywhere?

Accepted Answer

LayerCal is free and open source under the MIT licence. All calculations and code generation run in your browser, with no account and no backend. The AI Architecture Advisor is bring-your-own-key: your API key is stored in your browser and is sent only to the provider you choose, which is Google, OpenAI or Anthropic. You can turn off key storage entirely in the advanced settings.

Layer	Parameters	Notes
Embedding	V × E	Vocabulary size by embedding dimension
Linear	I × O + O	Bias term adds one per output unit
Conv2D	C_in × C_out × K² + C_out	Independent of input resolution
LSTM	4(IH + H² + 2H) × L × dir	Four gates, two bias vectors per gate
GRU	3(IH + H² + 2H) × L × dir	Three gates, so 75 percent of an LSTM
Transformer block	12d² + 13d	When d_ff equals 4d
Self-attention	4(d² + d)	Query, key, value and output projections
BatchNorm, LayerNorm	2F	One scale and one shift per feature
ReLU, Softmax, Dropout, Pooling	0	No learnable parameters

Mode	Bytes per parameter	Breakdown
Inference FP32	4	Weights only
Inference FP16 or BF16	2	Weights only
Inference INT8	1	Quantised weights
Training with Adam	16	Weights, gradients and two moment buffers, at any precision

Neural network parameter and memory reference

Parameter count by layer type

Memory per parameter

Frequently asked questions