filipriec/komp_ac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

import page error propagation

Browse Source
This commit is contained in:
Priec
2026-08-17 23:56:21 +02:00
parent 9b2b429dc6
commit b71ed4f42f
16 changed files with 496 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
web/locales/cs/main.ftl
View File

@@ -580,11 +580,16 @@ import-target-tables = Cílová tabulka
import-choose-table = Vyberte tabulku
import-csv-file = Soubor CSV
import-csv-data = Data CSV
import-csv-format-hint = První řádek je vlastní záhlaví souboru a každý další řádek jsou data. Použijte CSV v UTF-8 s oddělovačem čárka a každé pole uzavřete do dvojitých uvozovek, například: "hl","he","s"
import-csv-format-hint = První řádek se obvykle použije jako názvy sloupců souboru. Použijte CSV v UTF-8 s oddělovačem čárka a každé pole uzavřete do dvojitých uvozovek, například: "hl","he","s"
import-continue = Pokračovat
import-carried = Import do tabulky { $table } v rozsahu { $scope }.
import-error-title = CSV se nepodařilo importovat
import-import-rows = Importovat
import-failure-message = Před zastavením importu se importovalo { $inserted } z { $source_rows } řádků. Selhal řádek CSV { $row }.
Backend: { $error }
Již importované řádky zůstávají v tabulce. Aby nevznikly duplicity, neimportujte znovu celý soubor: nejprve tyto řádky odstraňte nebo importujte pouze chybný a zbývající řádky.
import-failure-no-rows-message = Nic se neimportovalo. Selhal řádek CSV { $row }.
Backend: { $error }
import-success-title = Import dokončen
import-success-message = Vloženo { $inserted ->
[one] { $inserted } řádek
@@ -597,14 +602,32 @@ import-success-message = Vloženo { $inserted ->
}.
# --- Krok 2: mapování ------------------------------------------------------
import-mapping-heading = Odkud bere hodnotu každý sloupec tabulky { $table }?
import-mapping-hint = Každý sloupec tabulky je uveden jednou, takže žádný z nich nelze naplnit dvakrát. Hodnoty se berou podle pozice: název zdrojového sloupce se zobrazuje proto, abyste jej rozeznali, a o ničem nerozhoduje. Sloupec ponechaný na Neimportovat se nezapíše a ostatní sloupce souboru se ignorují.
import-mapping-heading = Přiřaďte sloupce CSV do tabulky { $table }
import-mapping-hint = První řádek CSV byl načten jako názvy jeho sloupců. Připojte pouze sloupce CSV, které chcete importovat; nepřipojené sloupce tabulky zůstanou prázdné.
import-mapping-rows = Soubor obsahuje { $rows } datových řádků.
import-summary-mapped-short = namapováno
import-summary-attention-short = sloupců tabulky bez mapování
import-csv-columns = Nalezené sloupce CSV
import-csv-columns-hint = Přetáhněte sloupec na sloupec tabulky nebo na něj klikněte a potom klikněte na řádek tabulky. Malý text je první datová hodnota.
import-table-columns = Původní sloupce tabulky
import-table-columns-hint = Zobrazen je každý sloupec tabulky. Pokud jej záměrně nechcete importovat, ponechte řádek jako „Nenamapováno“.
import-drag-source = Namapovat sloupec CSV { $column }
import-destination-unmapped = Nenamapováno — ponechat prázdné
import-value-from = Bere hodnoty z
import-clear-mapping = Zrušit
import-th-destination = Sloupec tabulky
import-th-required = Povinný
import-th-source = Bere hodnotu z
import-th-example = Ukázka
import-source-none = Neimportovat
import-th-status = Stav
import-source-none = Přeskočit tento sloupec souboru
import-source-column = Sloupec { $position }
import-required-short = povinný
import-status-mapped = Namapováno
import-status-duplicate = Stejný název je použit dvakrát
import-status-unmapped = Nenamapováno
import-status-near = Podobá se na
import-status-near-match = Podobá se na { $column }
import-source-option = { $name } — pozice { $position }
import-source-unnamed = pozice { $position }
import-back-to-source = Zpět na soubor

31
web/locales/en/main.ftl
View File

@@ -570,11 +570,16 @@ import-target-tables = Target table
import-choose-table = Choose a table
import-csv-file = CSV file
import-csv-data = CSV data
import-csv-format-hint = The first row is the file's own header, and every row after it is data. Use UTF-8 CSV with a comma separator and every field in double quotes, for example: "hl","he","s"
import-csv-format-hint = The first row is normally used as the file's column names. Use UTF-8 CSV with a comma separator and every field in double quotes, for example: "hl","he","s"
import-continue = Continue
import-carried = Importing into { $table } in { $scope }.
import-error-title = Could not import CSV
import-import-rows = Import
import-failure-message = Imported { $inserted } of { $source_rows } rows before the import stopped. CSV row { $row } failed.
Backend: { $error }
Rows already imported remain in the table. To avoid duplicates, do not retry the whole file: remove those rows first, or import only the failed and remaining rows.
import-failure-no-rows-message = Nothing was imported. CSV row { $row } failed.
Backend: { $error }
import-success-title = Import complete
import-success-message = Inserted { $inserted ->
[one] { $inserted } row
@@ -585,14 +590,32 @@ import-success-message = Inserted { $inserted ->
}.
# --- Step 2: mapping -------------------------------------------------------
import-mapping-heading = Where does each column of { $table } take its value from?
import-mapping-hint = Every column of the table is listed once, so none of them can be filled twice. Values are taken by position: a source column's name is shown so you can recognise it, and decides nothing. A column left on Do not import is not written, and the file's other columns are ignored.
import-mapping-heading = Map CSV columns into { $table }
import-mapping-hint = The CSV's first row was read as its column names. Connect only the CSV columns you want to import; table columns left without a connection stay empty.
import-mapping-rows = The file holds { $rows } data rows.
import-summary-mapped-short = mapped
import-summary-attention-short = table columns not mapped
import-csv-columns = CSV columns found
import-csv-columns-hint = Drag one onto a table column, or click it and then click a table row. The small text is the first data value.
import-table-columns = Original table columns
import-table-columns-hint = Every table column is shown. Leave a row as “Not mapped” when you intentionally do not want to import it.
import-drag-source = Map CSV column { $column }
import-destination-unmapped = Not mapped — leave empty
import-value-from = Gets values from
import-clear-mapping = Clear
import-th-destination = Table column
import-th-required = Required
import-th-source = Takes its value from
import-th-example = Example
import-source-none = Do not import
import-th-status = Status
import-source-none = Skip this file column
import-source-column = Column { $position }
import-required-short = required
import-status-mapped = Mapped
import-status-duplicate = Same name appears twice
import-status-unmapped = Not mapped
import-status-near = Looks like
import-status-near-match = Looks like { $column }
import-source-option = { $name } — position { $position }
import-source-unnamed = position { $position }
import-back-to-source = Back to the file

31
web/locales/sk/main.ftl
View File

@@ -580,11 +580,16 @@ import-target-tables = Cieľová tabuľka
import-choose-table = Vyberte tabuľku
import-csv-file = Súbor CSV
import-csv-data = Údaje CSV
import-csv-format-hint = Prvý riadok je vlastná hlavička súboru a každý ďalší riadok sú údaje. Použite CSV v UTF-8 s oddeľovačom čiarka a každé pole uzavrite do dvojitých úvodzoviek, napríklad: "hl","he","s"
import-csv-format-hint = Prvý riadok sa zvyčajne použije ako názvy stĺpcov súboru. Použite CSV v UTF-8 s oddeľovačom čiarka a každé pole uzavrite do dvojitých úvodzoviek, napríklad: "hl","he","s"
import-continue = Pokračovať
import-carried = Import do tabuľky { $table } v rozsahu { $scope }.
import-error-title = CSV sa nepodarilo importovať
import-import-rows = Importovať
import-failure-message = Pred zastavením importu sa importovalo { $inserted } z { $source_rows } riadkov. Zlyhal riadok CSV { $row }.
Backend: { $error }
Už importované riadky zostávajú v tabuľke. Aby nevznikli duplicity, neimportujte znova celý súbor: najprv tieto riadky odstráňte alebo importujte iba chybný a zostávajúce riadky.
import-failure-no-rows-message = Nič sa neimportovalo. Zlyhal riadok CSV { $row }.
Backend: { $error }
import-success-title = Import dokončený
import-success-message = Vložený { $inserted ->
[one] { $inserted } riadok
@@ -595,14 +600,32 @@ import-success-message = Vložený { $inserted ->
}.
# --- Krok 2: mapovanie -----------------------------------------------------
import-mapping-heading = Odkiaľ berie hodnotu každý stĺpec tabuľky { $table }?
import-mapping-hint = Každý stĺpec tabuľky je uvedený raz, takže žiadny z nich sa nedá naplniť dvakrát. Hodnoty sa berú podľa pozície: názov zdrojového stĺpca sa zobrazuje preto, aby ste ho rozoznali, a o ničom nerozhoduje. Stĺpec ponechaný na Neimportovať sa nezapíše a ostatné stĺpce súboru sa ignorujú.
import-mapping-heading = Priraďte stĺpce CSV do tabuľky { $table }
import-mapping-hint = Prvý riadok CSV bol načítaný ako názvy jeho stĺpcov. Pripojte iba stĺpce CSV, ktoré chcete importovať; nepripojené stĺpce tabuľky zostanú prázdne.
import-mapping-rows = Súbor obsahuje { $rows } riadkov údajov.
import-summary-mapped-short = namapovaných
import-summary-attention-short = stĺpcov tabuľky bez mapovania
import-csv-columns = Nájdené stĺpce CSV
import-csv-columns-hint = Potiahnite stĺpec na stĺpec tabuľky alebo naň kliknite a potom kliknite na riadok tabuľky. Malý text je prvá hodnota údajov.
import-table-columns = Pôvodné stĺpce tabuľky
import-table-columns-hint = Zobrazený je každý stĺpec tabuľky. Ak ho zámerne nechcete importovať, ponechajte riadok ako „Nenamapované“.
import-drag-source = Namapovať stĺpec CSV { $column }
import-destination-unmapped = Nenamapované — ponechať prázdne
import-value-from = Berie hodnoty z
import-clear-mapping = Zrušiť
import-th-destination = Stĺpec tabuľky
import-th-required = Povinný
import-th-source = Berie hodnotu z
import-th-example = Ukážka
import-source-none = Neimportovať
import-th-status = Stav
import-source-none = Preskočiť tento stĺpec súboru
import-source-column = Stĺpec { $position }
import-required-short = povinný
import-status-mapped = Namapované
import-status-duplicate = Rovnaký názov je použitý dvakrát
import-status-unmapped = Nenamapované
import-status-near = Podobá sa na
import-status-near-match = Podobá sa na { $column }
import-source-option = { $name } — pozícia { $position }
import-source-unnamed = pozícia { $position }
import-back-to-source = Späť na súbor

3
web/src/lib.rs
View File

@@ -19,6 +19,9 @@ mod i18n;
#[path = "../../common/src/system_column.rs"]
#[allow(dead_code)]
mod system_column;
#[path = "../../common/src/grpc_error.rs"]
#[allow(dead_code)]
mod grpc_error;
mod analytics {
include!(concat!(
env!("CARGO_MANIFEST_DIR"),

7
web/src/pages/import_export/import/destination.rs
View File

@@ -1,9 +1,8 @@
//! The columns an import may write into, and how the form names one.
//!
//! The mapping is read destination-first: every writable column of the table
//! gets one row, and the row asks where its value comes from. That is what
//! makes "one destination filled twice" impossible to express rather than
//! merely refused — each destination appears exactly once, by construction.
//! The review is source-first: every file position gets one row and chooses a
//! destination by stable identity. Duplicate destination choices are prevented
//! in the browser and refused again by the Rust mapping validation.
use crate::definitions::table_structure::{TableColumn, TableStructureResponse};

142
web/src/pages/import_export/import/logic.rs
View File

@@ -5,9 +5,8 @@ use axum::{
http::{HeaderMap, HeaderValue, header},
response::{Html, IntoResponse, Redirect, Response},
};
// The mapping posts `destination` and `source_position` once per destination
// row, and `axum::Form` (serde_urlencoded) cannot decode repeated keys into a
// `Vec`.
// The review posts a destination and selected CSV position once per table row;
// only axum-extra's HTML form decoder preserves repeated keys as vectors.
use axum_extra::extract::Form;
use crate::{
@@ -79,8 +78,8 @@ pub(crate) async fn source_step(
render_step(state, &headers, form, Step::Source).await
}
/// POST /admin/import/prepare — the mapping: one row per writable destination
/// column, asking where its value comes from.
/// POST /admin/import/prepare — parsed CSV columns mapped onto fixed table
/// columns.
pub(crate) async fn prepare_step(
State(state): State<AppState>,
headers: HeaderMap,
@@ -99,42 +98,59 @@ pub(crate) async fn prepare_step(
Err(message) => return reject(&headers, message),
};
// What the user has already answered, if they are coming back from the
// preview. Keyed by the destination's identity rather than by row order, so
// a table that gained or lost a column keeps the answers for the rest.
// Answers are keyed by stable destination id, so a rename between review
// and import does not retarget one.
let answered = form
.rows()
.into_iter()
.filter_map(|(key, position)| Some((key.to_string(), position?)))
.collect::<HashMap<_, _>>();
let has_answers = !form.destination.is_empty();
let positions_by_name = source
.header
.iter()
.enumerate()
.fold(HashMap::<&str, Vec<usize>>::new(), |mut positions, (index, name)| {
positions.entry(name.as_str()).or_default().push(index);
positions
});
let rows = destination
.columns
.iter()
.map(|column| {
let key = column.key.encode();
let chosen_index = if has_answers {
answered.get(&key).copied()
} else {
positions_by_name
.get(column.name.as_str())
.filter(|positions| positions.len() == 1)
.and_then(|positions| positions.first().copied())
};
MappingRow {
key,
name: column.name.clone(),
required: column.required,
chosen: chosen_index
.map(|index| (index + 1).to_string())
.unwrap_or_default(),
example: chosen_index
.map(|index| source.example(index).to_string())
.unwrap_or_default(),
}
})
.collect::<Vec<_>>();
let mapped = rows.iter().filter(|row| !row.chosen.is_empty()).count();
let attention = rows.len().saturating_sub(mapped);
let step = Step::Mapping(MappingStep {
table_name: destination.table_name.clone(),
rows: destination
.columns
.iter()
.map(|column| {
let key = column.key.encode();
// Nothing starts out mapped. An exact name match would be a
// reasonable guess, but a guess is what this page exists to
// avoid — every destination is the user's to answer.
let chosen = answered
.get(&key)
.copied()
.filter(|index| *index < source.width());
MappingRow {
key,
name: column.name.clone(),
required: column.required,
example: chosen
.map(|index| source.example(index).to_string())
.unwrap_or_default(),
chosen: chosen.map(|index| (index + 1).to_string()).unwrap_or_default(),
}
})
.collect(),
rows,
sources: source_options(locale, &source),
source_rows: source.rows.len(),
mapped,
attention,
});
render_step(state, &headers, form, step).await
}
@@ -224,7 +240,7 @@ pub(crate) async fn import_csv(
};
let mut inserted = 0usize;
for chunk in converted.chunks(1_000) {
for (chunk_index, chunk) in converted.chunks(1_000).enumerate() {
let request = PostTableDataBulkRequest {
profile_name: profile_name.clone(),
table_name: destination.table_name.clone(),
@@ -239,7 +255,15 @@ pub(crate) async fn import_csv(
.await
{
Ok(response) => response.into_inner(),
Err(error) => return grpc_error(&headers, &error),
Err(error) => {
return import_error(
&headers,
&error,
inserted,
chunk_index * 1_000,
prepared.row_count(),
);
}
};
inserted += response
.responses
@@ -361,7 +385,8 @@ async fn prepared(
) -> Result<(Destination, Source, Prepared), Response> {
let locale = Locale::from_headers(headers);
let destination = destination(state, headers, form).await?;
let source = read_source(locale, &form.csv_data).map_err(|message| reject(headers, message))?;
let source = read_source(locale, &form.csv_data)
.map_err(|message| reject(headers, message))?;
// Every posted destination has to still exist and still be writable. A key
// that resolves to nothing is refused rather than skipped: skipping it
@@ -396,6 +421,7 @@ fn source_options(locale: Locale, source: &Source) -> Vec<SourceOption> {
let name = source.name(index).trim();
SourceOption {
position: index + 1,
name: name.to_string(),
label: if name.is_empty() {
tr!(locale, "import-source-unnamed", "position" => position)
} else {
@@ -469,6 +495,51 @@ fn grpc_error(headers: &HeaderMap, error: &tonic::Status) -> Response {
.into_response(Locale::from_headers(headers), ui::render_error)
}
fn import_error(
headers: &HeaderMap,
error: &tonic::Status,
inserted_before_chunk: usize,
chunk_start: usize,
prepared_rows: usize,
) -> Response {
let Some((failed_row_index, inserted_in_chunk)) = bulk_failure(error) else {
return grpc_error(headers, error);
};
let locale = Locale::from_headers(headers);
let failure = crate::ui::FormError::from_status(error);
if matches!(failure, crate::ui::FormError::Unauthenticated) {
return failure.into_response(locale, ui::render_error);
}
let status = failure.status_code();
let message = ui::render_import_failure(
locale,
inserted_before_chunk + inserted_in_chunk,
prepared_rows,
// One header row precedes the data, and CSV rows are one-based.
chunk_start + failed_row_index + 2,
failure.message(),
);
(status, Html(message)).into_response()
}
fn bulk_failure(error: &tonic::Status) -> Option<(usize, usize)> {
let failed_row_index = error
.metadata()
.get(crate::grpc_error::BULK_FAILED_ROW_INDEX_METADATA_KEY)?
.to_str()
.ok()?
.parse()
.ok()?;
let inserted_rows = error
.metadata()
.get(crate::grpc_error::BULK_INSERTED_ROWS_METADATA_KEY)?
.to_str()
.ok()?
.parse()
.ok()?;
Some((failed_row_index, inserted_rows))
}
fn load_error(headers: &HeaderMap, error: LoadError) -> Response {
let locale = Locale::from_headers(headers);
error
@@ -502,10 +573,8 @@ mod tests {
let options = source_options(Locale::default(), &source());
assert_eq!(options.len(), 5);
assert_eq!(options[2].position, 3);
assert!(options[2].label.contains("hl"), "{}", options[2].label);
assert!(options[2].label.contains('3'), "{}", options[2].label);
assert_eq!(options[2].example, "value-a");
}
/// A file that left a column unnamed still has that column, and it is still
@@ -515,6 +584,5 @@ mod tests {
let options = source_options(Locale::default(), &source());
assert!(options[3].label.contains('4'), "{}", options[3].label);
assert_eq!(options[3].example, "value-b");
}
}

9
web/src/pages/import_export/import/prepare.rs
View File

@@ -1,11 +1,9 @@
//! Turning the uploaded file plus the user's mapping into the one CSV the
//! import understands.
//!
//! There is one shape of input: a header row, then data rows. The header is
//! read so the user can tell the file's columns apart, and is then done with —
//! it is never imported, and it never decides anything. What decides is the
//! mapping: destination column `a` takes source position 3, because the user
//! said so.
//! The first row names the CSV columns and every later row is data. The names
//! are displayed as draggable sources; what decides where values land is the
//! mapping: destination column `a` takes source position 3.
//!
//! Positions, not names, all the way through. A file may perfectly well have
//! two columns called `name`; the page shows them as `name — position 1` and
@@ -60,7 +58,6 @@ impl Source {
/// the same width.
pub(crate) fn read_source(locale: Locale, csv: &str) -> Result<Source, String> {
let mut rows = parse_csv(locale, csv)?;
// One row is a header with nothing under it, which is not an import.
if rows.len() < 2 {
return Err(tr!(locale, "import-err-no-data-rows"));
}

41
web/src/pages/import_export/import/state.rs
View File

@@ -16,17 +16,15 @@ pub(crate) struct ImportForm {
/// The uploaded file, header row and all.
#[serde(default)]
pub csv_data: String,
/// One entry per writable destination column, in the order the mapping step
/// renders them: the column's stable identity, not its name. See
/// One entry per table-column row: the destination's stable identity. See
/// [`DestinationKey`](super::destination::DestinationKey).
///
/// Posted once per row, which only `axum_extra`'s `Form` decodes into a
/// `Vec`.
#[serde(default)]
pub destination: Vec<String>,
/// The source position chosen for the destination at the same index, from
/// 1; empty means "do not import". The two vectors are filled in document
/// order, so they line up row for row.
/// The selected CSV position for the destination at the same index, from
/// 1; empty means intentionally not mapped.
#[serde(default)]
pub source_position: Vec<String>,
}
@@ -65,6 +63,7 @@ impl ImportForm {
self.destination
.iter()
.enumerate()
.filter(|(_, key)| !key.trim().is_empty())
.map(|(index, key)| {
let position = self
.source_position
@@ -90,32 +89,27 @@ pub(crate) enum Step {
pub(crate) struct MappingStep {
pub table_name: String,
/// One row per writable destination column. A destination cannot be filled
/// twice because it appears exactly once.
/// One fixed row per destination table column.
pub rows: Vec<MappingRow>,
/// The file's columns, as they are offered in every picker.
/// Columns parsed from the CSV's first row, available to connect to a
/// destination by dragging or selecting.
pub sources: Vec<SourceOption>,
pub source_rows: usize,
pub mapped: usize,
pub attention: usize,
}
/// One destination column, asking where its value comes from.
/// One destination table column and the source position connected to it.
pub(crate) struct MappingRow {
/// The stable identity, for the hidden field.
pub key: String,
/// The name the table shows, and the name the prepared header will use.
pub name: String,
pub required: bool,
/// The chosen source position as the form carries it — one-based, empty for
/// "do not import".
/// One-based source position, empty when intentionally not mapped.
pub chosen: String,
/// The first data row's value at the chosen position, so the row shows what
/// it is actually going to import.
pub example: String,
}
impl MappingRow {
/// Whether this row takes `option`'s position, for re-rendering the picker
/// with the user's own answer selected.
pub(crate) fn takes(&self, option: &SourceOption) -> bool {
self.chosen == option.position.to_string()
}
@@ -123,8 +117,8 @@ impl MappingRow {
/// One column of the uploaded file, as the pickers offer it.
pub(crate) struct SourceOption {
/// One-based, which is how the page counts and how the form posts.
pub position: usize,
pub name: String,
/// `hl — position 3`, or just the position when the file left the name
/// blank. Built in Rust so it is translated once.
pub label: String,
@@ -243,4 +237,15 @@ mod tests {
assert_eq!(form.rows(), vec![("id:1", Some(1)), ("id:2", None)]);
}
#[test]
fn an_ignored_source_row_has_no_destination_assignment() {
let form = ImportForm {
destination: vec![String::new(), "id:2".to_string()],
source_position: vec!["1".to_string(), "2".to_string()],
..ImportForm::default()
};
assert_eq!(form.rows(), vec![("id:2", Some(1))]);
}
}

98
web/src/pages/import_export/import/ui.rs
View File

@@ -47,6 +47,32 @@ pub(crate) fn render_error(locale: Locale, message: &str) -> String {
))
}
pub(crate) fn render_import_failure(
locale: Locale,
inserted: usize,
prepared_rows: usize,
csv_row: usize,
backend_message: &str,
) -> String {
let message_key = if inserted == 0 {
"import-failure-no-rows-message"
} else {
"import-failure-message"
};
render(&Alert::error(
locale,
&tr!(locale, "import-error-title"),
&tr!(
locale,
message_key,
"inserted" => inserted as i64,
"source_rows" => prepared_rows as i64,
"row" => csv_row as i64,
"error" => backend_message.to_string(),
),
))
}
pub(crate) fn render_success(
locale: Locale,
inserted: usize,
@@ -73,9 +99,7 @@ mod tests {
use crate::auth::AuthorizationSnapshot;
use crate::pages::import_export::common::loader::{Catalog, Profile};
use super::super::state::{
ImportForm, MappingRow, MappingStep, PreviewStep, SourceOption,
};
use super::super::state::{ImportForm, MappingRow, MappingStep, PreviewStep, SourceOption};
fn strings(values: &[&str]) -> Vec<String> {
values.iter().map(|value| value.to_string()).collect()
@@ -101,7 +125,7 @@ mod tests {
table_name: "customers".to_string(),
csv_data: "\"hl\",\"he\"\n\"value-a\",\"value-b\"\n".to_string(),
destination: strings(&["id:42", "id:57"]),
source_position: strings(&["1", ""]),
source_position: strings(&["1", "2"]),
},
step,
}
@@ -115,60 +139,64 @@ mod tests {
key: "id:42".to_string(),
name: "a".to_string(),
required: true,
chosen: "1".to_string(),
example: "value-a".to_string(),
chosen: "1".to_string(),
},
MappingRow {
key: "id:57".to_string(),
name: "b".to_string(),
required: false,
chosen: String::new(),
example: String::new(),
chosen: String::new(),
},
],
sources: vec![
SourceOption {
position: 1,
label: "hl \u{2014} position 1".to_string(),
name: "hl".to_string(),
label: "hl — position 1".to_string(),
example: "value-a".to_string(),
},
SourceOption {
position: 2,
label: "he \u{2014} position 2".to_string(),
name: "he".to_string(),
label: "he — position 2".to_string(),
example: "value-b".to_string(),
},
],
source_rows: 1,
mapped: 1,
attention: 1,
})
}
/// The first step asks for a destination and a file, and nothing else —
/// there is one shape of input, so there is nothing to choose about it.
/// The first step stays focused on the destination and file. The first-row
/// question comes after parsing, where the page can show the real values.
#[test]
fn the_first_step_asks_for_a_table_and_a_file() {
let html = render_page(&page(Step::Source));
assert!(html.contains(r#"action="/admin/import""#), "{html}");
assert!(html.contains(r#"name="table_name""#), "{html}");
assert!(html.contains(r#"name="csv_data""#), "{html}");
assert!(!html.contains("source_mode"), "{html}");
assert!(!html.contains(r#"name="header_mode""#), "{html}");
}
/// The mapping is read destination-first: one row per column of the table,
/// each asking where its value comes from. A destination cannot be filled
/// twice because it appears exactly once.
/// The table columns stay fixed while parsed CSV columns can be connected
/// to them or deliberately left unused.
#[test]
fn the_mapping_step_asks_every_destination_column_once() {
fn the_mapping_step_reviews_every_source_column_once() {
let html = render_step(&page(mapping()));
assert_eq!(html.matches(r#"name="source_position""#).count(), 2);
assert_eq!(html.matches(r#"name="destination""#).count(), 2);
assert!(html.contains("<code>a</code>"), "{html}");
assert!(html.contains("<code>b</code>"), "{html}");
// The file's columns are offered by name and position together.
assert!(html.contains("hl \u{2014} position 1"), "{html}");
assert!(html.contains("CSV columns found"), "{html}");
assert!(html.contains("Original table columns"), "{html}");
assert!(html.contains(r#"data-source-position="1""#), "{html}");
assert!(html.contains("value-a"), "{html}");
assert!(html.contains(r#"name="destination" value="id:42""#), "{html}");
assert!(html.contains(r#"<option value="1" data-example="value-a" selected>"#), "{html}");
// Nothing is mapped unless the user said so.
assert!(html.contains(">Do not import</option>"), "{html}");
assert!(html.contains(">Not mapped — leave empty</option>"), "{html}");
}
/// The destination travels as its stable identity, never as its name, so a
@@ -202,10 +230,38 @@ mod tests {
assert!(html.contains("aa \u{2014} position 6"), "{html}");
assert!(html.contains("number"), "{html}");
assert!(html.contains(r#"hx-post="/admin/import""#), "{html}");
assert!(
html.contains(r#"formaction="/admin/import/prepared.csv""#),
"{html}"
);
assert!(html.contains("&#34;a&#34;,&#34;b&#34;,&#34;c&#34;"), "{html}");
// The mapping travels with it, so the download and the import prepare
// the identical file.
assert!(html.contains(r#"name="destination" value="id:42""#), "{html}");
assert!(html.contains(r#"name="source_position" value="1""#), "{html}");
}
#[test]
fn an_import_failure_explains_partial_progress_and_the_backend_error() {
let html = render_import_failure(
Locale::English,
599,
1_200,
601,
"Internal server error (reference: example-id)",
);
assert!(html.contains("Imported 599 of 1200 rows"), "{html}");
assert!(html.contains("CSV row 601 failed"), "{html}");
assert!(html.contains("reference: example-id"), "{html}");
assert!(html.contains("Rows already imported remain"), "{html}");
}
#[test]
fn a_first_row_failure_does_not_claim_that_rows_remain() {
let html = render_import_failure(Locale::English, 0, 20, 2, "Invalid value");
assert!(html.contains("Nothing was imported"), "{html}");
assert!(!html.contains("already imported remain"), "{html}");
}
}

49
web/static/app.css
View File

@@ -352,13 +352,44 @@
/* ---------- Import preparation (pages/import_export/import) ---------- */
/* A mapping row is read left to right as a sentence: the column `a` takes its
value from "hl — position 3", which on the first row holds "value-a". The
example is what people actually recognise their data by, so it gets the
room, and it is the last thing on the line because it is the confirmation
rather than the choice. */
.mapping-table select { width: 100%; min-width: 220px; }
.mapping-table .example { max-width: 320px; overflow: hidden; text-overflow: ellipsis; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; color: #33415c; }
/* CSV columns are a movable palette; table columns stay fixed as the import
destinations. A select in every row is the keyboard/touch equivalent of
dragging a source chip onto it. */
.mapping-summary { display: flex; flex-wrap: wrap; align-items: center; gap: 8px 16px; margin: 14px 0; padding: 10px 12px; border-radius: 8px; background: #f5f7fa; }
.mapping-summary strong { color: #24324a; font-variant-numeric: tabular-nums; }
.mapping-workbench { display: grid; grid-template-columns: minmax(230px, .75fr) minmax(480px, 1.6fr); align-items: start; gap: 14px; }
.mapping-workbench h3 { margin: 0; color: #24324a; font-size: 14px; }
.mapping-workbench h3 + .hint { margin: 4px 0 12px; font-size: 12px; }
.source-palette, .destination-map { padding: 14px; border: 1px solid #dfe5ed; border-radius: 9px; background: #fafbfc; }
.source-chips { display: grid; gap: 7px; }
.source-chip { position: relative; display: grid; grid-template-columns: minmax(0, 1fr) auto; align-items: center; gap: 8px; width: 100%; padding: 9px 10px; border: 1px solid #cfd9e6; border-radius: 7px; color: #33415c; background: white; text-align: left; cursor: grab; transition: border-color 120ms ease, box-shadow 120ms ease, opacity 120ms ease; }
.source-chip:hover, .source-chip.is-selected { border-color: #78a1df; box-shadow: 0 3px 10px rgb(31 43 58 / 9%); }
.source-chip.is-selected { outline: 3px solid rgb(37 99 235 / 12%); }
.source-chip.is-mapped { border-color: #a8cfb3; background: #f4fbf6; }
.source-chip.is-dragging { opacity: .55; cursor: grabbing; }
.source-chip code { overflow: hidden; text-overflow: ellipsis; }
.source-chip small { max-width: 100px; overflow: hidden; color: #778396; text-overflow: ellipsis; white-space: nowrap; }
.destination-rows { display: grid; gap: 7px; margin: 0; padding: 0; list-style: none; }
.destination-rows li { display: grid; grid-template-columns: minmax(100px, .7fr) 22px minmax(210px, 1.3fr) minmax(70px, .65fr) auto; align-items: center; gap: 8px; min-height: 62px; padding: 8px 9px; border: 1px dashed #cdd6e2; border-radius: 7px; background: white; transition: border-color 120ms ease, background 120ms ease, box-shadow 120ms ease; }
.destination-rows li.is-mapped { border-style: solid; border-color: #b9cce8; }
.destination-rows li.drop-ready { border-style: solid; border-color: #2563eb; background: #edf4ff; box-shadow: 0 0 0 3px rgb(37 99 235 / 12%); }
.destination-rows li.just-mapped { background: #eef8f1; }
.destination-name { display: flex; flex-wrap: wrap; align-items: center; gap: 5px; min-width: 0; }
.destination-name code { overflow: hidden; color: #24324a; text-overflow: ellipsis; }
.mapping-arrow { color: #9aa5b4; text-align: center; }
.destination-rows label { display: grid; gap: 3px; color: #778396; font-size: 10px; letter-spacing: .04em; text-transform: uppercase; }
.destination-rows select { min-width: 0; color: #24324a; font-size: 13px; letter-spacing: normal; text-transform: none; }
.mapped-example { min-width: 0; overflow: hidden; color: #667385; font: 11px/1.4 ui-monospace, monospace; text-overflow: ellipsis; white-space: nowrap; }
.clear-mapping { padding: 6px 8px; border: 1px solid #d6dde7; border-radius: 6px; color: #59677a; background: white; font-size: 11px; cursor: pointer; }
.clear-mapping:hover:not(:disabled) { color: #a12b2b; background: #fdf3f3; }
.clear-mapping:disabled { color: #bdc4ce; background: #f7f8fa; cursor: default; }
@media (max-width: 850px) {
.mapping-workbench { grid-template-columns: 1fr; }
.source-chips { grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); }
.destination-rows li { grid-template-columns: minmax(90px, .7fr) 18px minmax(180px, 1.3fr) auto; }
.mapped-example { display: none; }
}
/* A secondary action in a form's action row — "back a step". The row's own
rule paints every button as the primary one, so this has to say otherwise
@@ -380,6 +411,10 @@
}
@media (max-width: 520px) {
.destination-rows li { grid-template-columns: minmax(0, 1fr) auto; }
.destination-rows .mapping-arrow { display: none; }
.destination-rows label { grid-column: 1 / -1; }
.destination-rows .clear-mapping { grid-column: 2; grid-row: 1; }
.alias-list-head { display: none; }
.alias-row { grid-template-columns: minmax(0, 1fr) 18px minmax(0, 1.3fr); }
.alias-row > button { grid-column: 3; width: 100%; }

107
web/templates/pages/import_export/import/behaviour.html
View File

@@ -1,34 +1,111 @@
{#
Two things the mapping table does in the browser, both of them conveniences
over rules the server enforces anyway.
Mapping conveniences over rules the server enforces anyway.
Everything is delegated from `document`, because each step swaps the whole
block: a listener bound to a select would be thrown away with it.
#}
<script>
(function () {
function sync() {
const selects = Array.from(document.querySelectorAll("[data-source-select]"));
// A source column can fill one destination only. The mapping refuses a
// repeat outright; this stops it being offered, so the refusal is not
// how the user finds out.
let selectedSource = "";
function sync(workbench) {
if (!workbench) return;
const rows = Array.from(workbench.querySelectorAll("[data-map-target]"));
const selects = rows.map((row) => row.querySelector("[data-source-select]"));
const taken = new Set(selects.map((select) => select.value).filter(Boolean));
for (const select of selects) {
for (const option of select.options) {
option.disabled = option.value !== "" && option.value !== select.value && taken.has(option.value);
}
// The example beside the row is the first data row's value at the
// chosen position, so the row shows what it is actually going to
// import rather than only which column it named.
const cell = select.closest("tr")?.querySelector("[data-example-cell]");
if (cell) cell.textContent = select.selectedOptions[0]?.dataset.example ?? "";
}
rows.forEach((row, index) => {
const option = selects[index].selectedOptions[0];
row.classList.toggle("is-mapped", Boolean(selects[index].value));
row.querySelector("[data-example-cell]").textContent = option?.dataset.example ?? "";
row.querySelector("[data-clear-mapping]").disabled = !selects[index].value;
});
workbench.querySelectorAll("[data-source-chip]").forEach((chip) => {
chip.classList.toggle("is-mapped", taken.has(chip.dataset.sourcePosition));
chip.classList.toggle("is-selected", chip.dataset.sourcePosition === selectedSource);
});
const summary = document.querySelector("[data-mapping-summary]");
if (summary) {
const mapped = taken.size;
summary.querySelector("[data-mapped-count]").textContent = mapped;
summary.querySelector("[data-attention-count]").textContent = rows.length - mapped;
}
}
function assign(workbench, position, row) {
if (!position || !row) return;
workbench.querySelectorAll("[data-source-select]").forEach((select) => {
if (select.value === position) select.value = "";
});
row.querySelector("[data-source-select]").value = position;
selectedSource = "";
row.classList.add("just-mapped");
setTimeout(function () { row.classList.remove("just-mapped"); }, 320);
sync(workbench);
}
document.addEventListener("change", function (event) {
if (event.target.matches("[data-source-select]")) sync();
if (event.target.matches("[data-source-select]")) {
sync(event.target.closest("[data-mapping-workbench]"));
}
});
document.body.addEventListener("htmx:afterSwap", sync);
sync();
document.addEventListener("click", function (event) {
const chip = event.target.closest("[data-source-chip]");
if (chip) {
selectedSource = selectedSource === chip.dataset.sourcePosition ? "" : chip.dataset.sourcePosition;
sync(chip.closest("[data-mapping-workbench]"));
return;
}
const clear = event.target.closest("[data-clear-mapping]");
if (clear) {
clear.closest("[data-map-target]").querySelector("[data-source-select]").value = "";
sync(clear.closest("[data-mapping-workbench]"));
return;
}
const row = event.target.closest("[data-map-target]");
if (row && selectedSource && !event.target.closest("select")) {
assign(row.closest("[data-mapping-workbench]"), selectedSource, row);
}
});
document.addEventListener("dragstart", function (event) {
const chip = event.target.closest("[data-source-chip]");
if (!chip) return;
event.dataTransfer.setData("text/plain", chip.dataset.sourcePosition);
event.dataTransfer.effectAllowed = "move";
chip.classList.add("is-dragging");
});
document.addEventListener("dragover", function (event) {
const row = event.target.closest("[data-map-target]");
if (!row) return;
event.preventDefault();
row.classList.add("drop-ready");
event.dataTransfer.dropEffect = "move";
});
document.addEventListener("dragleave", function (event) {
const row = event.target.closest("[data-map-target]");
if (row && !row.contains(event.relatedTarget)) row.classList.remove("drop-ready");
});
document.addEventListener("drop", function (event) {
const row = event.target.closest("[data-map-target]");
if (!row) return;
event.preventDefault();
row.classList.remove("drop-ready");
assign(row.closest("[data-mapping-workbench]"), event.dataTransfer.getData("text/plain"), row);
});
document.addEventListener("dragend", function (event) {
event.target.closest("[data-source-chip]")?.classList.remove("is-dragging");
document.querySelectorAll("[data-map-target].drop-ready").forEach((row) => row.classList.remove("drop-ready"));
});
document.body.addEventListener("htmx:afterSwap", function (event) {
selectedSource = "";
sync(event.detail.target.querySelector?.("[data-mapping-workbench]"));
});
sync(document.querySelector("[data-mapping-workbench]"));
})();
</script>

8
web/templates/pages/import_export/import/import.html
View File

@@ -12,11 +12,11 @@
than the form itself, so the scope, the table and the CSV the user chose stay
where they are while they move back and forth.
Its own `action` is the prepared-CSV download: a file has to come back from a
real browser submit, so the plain submit button is the one that downloads and
every other button carries an `hx-post` of its own.
Import is the form's default action, including an implicit submit with Enter.
Download overrides the action on its own button, so only that explicit click
can return a file.
#}
<form id="import-form" method="post" action="/admin/import/prepared.csv"
<form id="import-form" method="post" action="/admin/import"
hx-target="#import-step" hx-swap="outerHTML" hx-disabled-elt="this">
{% include "pages/import_export/import/step.html" %}
</form>

94
web/templates/pages/import_export/import/step.html
View File

@@ -38,6 +38,8 @@
<label class="wide">{{ nav.tr("import-csv-data") }}<textarea id="csv-data" name="csv_data" rows="12">{{ page.form.csv_data }}</textarea></label>
<small class="wide">{{ nav.tr("import-csv-format-hint") }}</small>
</div>
{% for key in page.form.destination %}<input type="hidden" name="destination" value="{{ key }}">{% endfor %}
{% for position in page.form.source_position %}<input type="hidden" name="source_position" value="{{ position }}">{% endfor %}
<div class="form-actions">
<a href="/">{{ nav.tr("common-cancel") }}</a>
<button type="button" hx-post="/admin/import/prepare" hx-include="closest form">{{ nav.tr("import-continue") }}</button>
@@ -53,40 +55,59 @@
<div class="builder-section">
<h2>{{ nav.tr_args("import-mapping-heading", [("table", step.table_name.clone())]) }}</h2>
<p class="hint">{{ nav.tr("import-mapping-hint") }}</p>
<p class="hint">{{ nav.tr_args("import-mapping-rows", [("rows", step.source_rows.to_string())]) }}</p>
<table class="builder-table mapping-table">
<thead>
<tr>
<th>{{ nav.tr("import-th-destination") }}</th>
<th>{{ nav.tr("import-th-required") }}</th>
<th>{{ nav.tr("import-th-source") }}</th>
<th>{{ nav.tr("import-th-example") }}</th>
</tr>
</thead>
<tbody>
{% for row in step.rows %}
<tr>
<td><code>{{ row.name }}</code></td>
<td>{% if row.required %}{{ nav.tr("common-yes") }}{% else %}<span class="hint">{{ nav.tr("common-no") }}</span>{% endif %}</td>
<td>
{#
The destination travels as its stable identity, so a rename
between here and Import moves the mapping with the column
instead of leaving it pointing at a name.
#}
<div class="mapping-summary" data-mapping-summary
data-mapped-label="{{ nav.tr("import-summary-mapped-short") }}"
data-attention-label="{{ nav.tr("import-summary-attention-short") }}">
<span><strong data-mapped-count>{{ step.mapped }}</strong> {{ nav.tr("import-summary-mapped-short") }}</span>
<span><strong data-attention-count>{{ step.attention }}</strong> {{ nav.tr("import-summary-attention-short") }}</span>
<span class="hint">{{ nav.tr_args("import-mapping-rows", [("rows", step.source_rows.to_string())]) }}</span>
</div>
<div class="mapping-workbench" data-mapping-workbench
data-unmapped-label="{{ nav.tr("import-destination-unmapped") }}">
<section class="source-palette">
<h3>{{ nav.tr("import-csv-columns") }}</h3>
<p class="hint">{{ nav.tr("import-csv-columns-hint") }}</p>
<div class="source-chips">
{% for source in step.sources %}
<button type="button" class="source-chip" draggable="true"
data-source-chip data-source-position="{{ source.position }}"
aria-label="{{ nav.tr_args("import-drag-source", [("column", source.label.clone())]) }}">
<code>{% if source.name.is_empty() %}{{ nav.tr_args("import-source-column", [("position", source.position.to_string())]) }}{% else %}{{ source.name }}{% endif %}</code>
<small>{{ source.example }}</small>
</button>
{% endfor %}
</div>
</section>
<section class="destination-map">
<h3>{{ nav.tr("import-table-columns") }}</h3>
<p class="hint">{{ nav.tr("import-table-columns-hint") }}</p>
<ol class="destination-rows">
{% for row in step.rows %}
<li data-map-target>
<input type="hidden" name="destination" value="{{ row.key }}">
<select name="source_position" data-source-select>
<option value="">{{ nav.tr("import-source-none") }}</option>
{% for option in step.sources %}
<option value="{{ option.position }}" data-example="{{ option.example }}"{% if row.takes(option) %} selected{% endif %}>{{ option.label }}</option>
{% endfor %}
</select>
</td>
<td class="example" data-example-cell>{{ row.example }}</td>
</tr>
{% endfor %}
</tbody>
</table>
<div class="destination-name">
<code>{{ row.name }}</code>
{% if row.required %}<span class="tag">{{ nav.tr("import-required-short") }}</span>{% endif %}
</div>
<span class="mapping-arrow" aria-hidden="true">←</span>
<label>
<span>{{ nav.tr("import-value-from") }}</span>
<select name="source_position" data-source-select>
<option value="">{{ nav.tr("import-destination-unmapped") }}</option>
{% for source in step.sources %}
<option value="{{ source.position }}" data-example="{{ source.example }}"{% if row.takes(source) %} selected{% endif %}>{{ source.label }}</option>
{% endfor %}
</select>
</label>
<span class="mapped-example" data-example-cell>{{ row.example }}</span>
<button type="button" class="clear-mapping" data-clear-mapping{% if row.chosen.is_empty() %} disabled{% endif %}>{{ nav.tr("import-clear-mapping") }}</button>
</li>
{% endfor %}
</ol>
</section>
</div>
</div>
<div class="form-actions">
<button type="button" class="secondary" hx-post="/admin/import/source" hx-include="closest form">{{ nav.tr("import-back-to-source") }}</button>
@@ -150,10 +171,11 @@
</div>
<div class="form-actions">
<button type="button" class="secondary" hx-post="/admin/import/prepare" hx-include="closest form">{{ nav.tr("import-back-to-mapping") }}</button>
{# A real submit: the answer is a file. #}
<button type="submit" formnovalidate>{{ nav.tr("import-download-prepared") }}</button>
<button type="button" hx-post="/admin/import" hx-include="closest form"
<button type="submit" hx-post="/admin/import" hx-include="closest form"
hx-target="#submission-status" hx-swap="innerHTML">{{ nav.tr("import-import-rows") }}</button>
{# A real submit: the answer is a file, and only this button asks for it. #}
<button type="submit" class="secondary" formaction="/admin/import/prepared.csv"
formnovalidate>{{ nav.tr("import-download-prepared") }}</button>
</div>
{% endmatch %}

4
web/templates/pages/import_export/table_picker.html
View File

@@ -6,8 +6,8 @@
keyboard. With scripting off the dropdown lists every table it was rendered
with, and the server still refuses a table that is not in the scope.
Bound again after every HTMX swap, because "Normalize headers" replaces the
fields — and with them both selects — with fresh nodes.
Bound again after every HTMX step swap, which replaces both selects with
fresh nodes when the user returns to the source step.
#}
<script>
(function () {